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SUMMARY  OF  CHANGES  TO  THE 
PART  GUIDANCE  FOR  2007 


Each  year,  the  Office  of  Management  and  Budget  (OMB)  reviews  the  guidance  for  completing 
the  PART  to  determine  if  any  updates  or  clarifications  are  necessary.  Only  minor  changes  have 
been  made  to  the  guidance  this  year. 


•  The  option  for  an  abbreviated  reassessmen  t  has  been  eliminated.  A  program  reassessment 
must  have  updated  responses  for  all  questions. 

•  The  guidance  on  efficiency  measures  has  been  expanded  to  accoun  t  for  differen  t  types  of 
measures  that  can  be  useful  for  managing  and  meet  the  PART  requirements.  These 
changes  were  made  in  response  to  issues  raised  during  the  2006  consistency  check  and 
appeals  process. 


2007  PART  SCHEDULE 


OMB  RMOs  save  completed  Attachment  A  to  OMB  shared  drive . January  19 

PARTWeb  available  for  data  entry . January  22 

Revised  guidance  on  answering  PART  questions  available . January  29 

Introductory  PART  Training . February  13  and  15 

Refresher  PART  Training . February  14 

PARTWeb  Training . February  21 

Agencies  give  OMB  RMOs  access  to  complete  PART  draft . March  30 

PARTs  ready  for  consistency  check . April  27 

Consistency  check  and  review  of  performance  measures . April  30  -  May  4 

OMB  RMOs  receive  feedback  on  consistency  check  and  performance  measures . May  9 

OMB  RMOs  revise  PARTs,  as  necessary,  to  address  consistency  issues . May  18 

OMB  RMOs  pass  back  PARTs  to  Agencies . May  18 

Agencies  submit  PART  Appeals . May  25 

OMB  RMOs  and  Agencies  work  out  issues  . May  28  -  June  1 

Agency  Spring  Update  of  PARTs  completed  in  previous  years . June  1  -  June  29 

Appeals  board  meets . June  1 1 

Appeals  board  provides  decisions . June  12 

PARTs  updated  to  reflect  appeals  board  decisions . July  9 


RMOs  complete  draft  summaries,  including  improvement  plans  on  new  PARTs . July  9 


PARTWeb  data  entry  locked 


vii 


August  3 


OVERVIEW  OF  THE  PART  PROCESS 


The  Program  Assessment  Rating  Tool  (PART)  is  a  diagnostic  tool  used  to  assess  the 
performance  of  Federal  programs  and  to  drive  improvements  in  program  performance.  Once 
completed,  PART  reviews  help  inform  budget  decisions  and  identify  actions  to  improve  results. 
Agencies  are  held  accountable  for  implementing  PART  follow-up  actions,  also  known  as 
improvement  plans,  for  each  of  their  programs. 

The  PART  is  designed  to  provide  a  consistent  approach  to  assessing  and  rating  programs  across 
the  Federal  government.  PART  assessments  review  overall  program  effectiveness,  from  how 
well  a  program  is  designed  to  how  well  it  is  implemented  and  what  results  it  achieves.  As  such, 
the  PART  examines  factors  that  the  program  or  agency  may  not  directly  control  but  may  be  able 
to  influence.  For  example,  if  a  PART  assessment  identifies  a  statutory  provision  that  impedes 
effectiveness,  one  of  the  follow-up  actions  may  be  for  the  agency  to  develop  and  propose 
legislative  changes  to  fix  it. 

The  PART  questions  are  generally  written  in  a  Yes/No  format.  A  Yes  answer  must  be  definite  and 
reflect  a  high  standard  of  performance.  Not  Applicable  may  be  an  appropriate  answer  when 
indicated  in  this  guidance;  Small  Extent  and  Large  Extent  are  options  for  the  questions  in  Section 
IV. 

Each  question  requires  a  clear  explanation  of  the  answer  and  citations  of  relevant  supporting 
evidence,  such  as  agency  performance  information,  independent  evaluations,  and  financial 
information.  Responses  must  be  evidence-based  and  not  rely  on  impressions  or  generalities. 

The  PART  is  central  to  the  Administration’s  Budget  and  Performance  Integration  (BPI)  Initiative 
because  it  drives  a  sustained  focus  on  results.  To  earn  a  high  PART  rating,  a  program  must  use 
performance  data  to  manage,  justify  its  resource  requests  based  on  the  performance  it  expects  to 
achieve,  and  continually  improve  efficiency  -  all  goals  of  the  BPI  Initiative.  A  list  of  the  criteria 
for  the  BPI  Initiative  can  be  found  in  Appendix  A. 

Completed  PARTs  are  available  for  public  scrutiny  at  www.ExpectMore.gov.  Each  program  that 
has  been  reviewed  using  the  PART  has  a  summary  on  ExpectMore.gov,  including  key  findings 
and  the  improvement  plan.  Every  summary  links  to  the  detailed  program  assessment  and  the 
program’s  website.  Because  PART  information  is  easily  accessible  and  searchable  on 
ExpectMore.gov,  it  is  more  transparent.  By  increasing  the  transparency  of  the  PART  process,  the 
Administration’s  aim  is  to  make  us  all  more  accountable  for  improving  program  performance 
every  year. 

The  PART  process  is  initiated  early  in  the  calendar  year.  Key  deadlines  are  described  below.  The 
schedule  for  completion  of  new  PART  reviews  and  reassessments  in  2007  is  on  page  vii  of  this 
document.  A  description  of  the  deadlines  for  2007,  training  information,  etc.  can  be  seen  online 
(http://www.whitehouse.gov/omb/part/fv2007/2007  guidance.pdf). 
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Additional  guidance  will  be  issued  throughout  the  year  as  needed. 


New  PART  and  Reassessment  Schedule  Overview  (replicated  from  PART 
Guidance  No.  2007-1) 

Initiate  2007  PARTs  -  OMB  and  Agencies  should  have  by  now  agreed  to  a  list  of  programs  to  be 
assessed  or  reassessed  this  year,  and  Agencies  should  initiate  work  on  those  PARTs.  PARTWeb 
for  2007  assessments  will  be  available  on  January  22.  PARTWeb  access  is  provided  using  a 
PARTWeb  ID.  Requests  to  establish  an  ID  are  made  through  the  following  link: 
https://max.omb.gov/maxportal/register.jsp. 

PART  training  -  OMB  will  offer  introductory  PART  training  on  February  13  and  15;  a  refresher 
course  on  February  14;  and  PARTWeb  training  on  February  21.  A  description  of  each  of  the 
courses  is  available  at  (http://www.whitehouse.gov/omb/part/fy2007/2007  guidance.pdf).  To 
register  for  training,  visit  (http://www.cio.gov/registration/bpi  070113/). 

By  Friday,  March  30,  Agencies  complete  the  first  PARTWeb  draft  of  all  2007  PARTs  for  OMB 
review.  The  draft  must  have  all  the  questions  answered  along  with  explanations  and  evidence,  as 
well  as  include  the  proposed  performance  measures  and  data.  As  it  was  last  year,  meeting  this 
deadline  will  be  a  deliverable  for  the  Budget  and  Performance  Integration  Initiative  and  will  be 
considered  when  determining  an  agency’s  progress  score  for  the  second  quarter  of  FY  2007. 

By  Friday,  April  27,  OMB  reviews  2007  PARTs  and  ensure  they  are  ready  for  a  consistency 
check  and  review  of  performance  measures.  PARTs  should  be  substantially  complete  and  meet 
all  requirements  of  the  guidance.  For  instance,  for  questions  with  multiple  elements,  the  answer 
must  address  all  elements  if  a  yes  is  given.  To  earn  a  “yes”  to  question  3.4,  an  efficiency  measure 
with  a  baseline  and  targets  should  be  listed  in  the  measures  section.  Likewise,  the  answers  to 
questions  4.1  -  4.3  should  be  consistent  with  the  data  provided  in  the  measures  section. 

By  May  11,  with  consistency  check  input,  OMB  revises  PARTs  in  response  to  consistency  check 
and  passback  to  Agencies  -  As  necessary,  PART  answers,  explanation,  and  evidence  are  revised 
to  respond  to  issues  identified  in  the  consistency  check.  In  some  cases,  performance  measures 
may  also  need  to  be  modified. 

Passback  and  appeals  process.  OMB  RMOs  and  Agencies  should  be  working  collaboratively  to 
complete  PARTs,  RMOs  should  passback  a  final  version  of  each  PART  to  agencies  by  Friday, 
May  18.  Following  passback  of  PARTs,  RMOs  and  Agencies  should  work  to  resolve  open 
issues.  If  any  disagreements  remain,  Agencies  have  the  opportunity  to  submit  appeals  on 
individual  PART  questions  to  a  high  level  appeals  board  by  Friday,  May  25.  The  appeals 
process  will  conclude  on  June  12.  Additional  guidance  on  the  appeals  process  will  be  provided 
later. 

By  June  25,  RMOs  complete  PART  revisions  to  reflect  appeals  board  decisions  -  As  necessary, 
PART  explanations  and  evidence  are  revised  to  reflect  appeals  board  decisions.  . 
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By  July  9,  Agencies  complete  the  first  draft  PART  summaries.  OMB  RMOs  and  Agencies  should 
begin  drafting  PART  summaries  shortly  after  PARTs  are  finalized  and  are  encouraged  to 
passback  draft  summaries  along  with  PARTs.  During  the  process  of  the  developing  the 
summaries,  the  Agency  and  RMO  should  agree  to  the  PART  improvement  plan. 

By  Friday,  August  3,  RMOs  finalize  all  PART  data,  including  data  related  to  the  Spring  Update 
of  PARTs  completed  in  prior  years.  PARTs  will  be  published  on  ExpectMore.gov  in  mid- 
August. 

Participants 

PART  is  a  collaborative  process  involving  both  agencies  and  OMB  participants.  Agencies 
contribute  a  variety  of  personnel  from  both  the  program  being  reviewed  and  from  the  central 
Department  offices.  These  typically  include: 

•  Program  staff  who  work  on  day-to-day  operations 

•  Budget  staff  who  work  on  resource  requests  and  justifications 

•  Planning/performance  staff  who  monitor  program  results  and  impacts 

The  primary  OMB  contact  will  be  the  program  examiner.  Other  OMB  staff  may  participate  as 
needed,  including  the  Office  of  Information  and  Regulatory  Affairs  (OIRA)  desk  officers  for 
regulatory  and  evaluation  questions,  Office  of  Federal  Financial  Management  (OFFM)  staff  for 
questions  on  financial  management,  etc. 

PARTs  are  completed  using  an  online  application  called  PARTWeb.  Through  PARTWeb, 
agencies  enter  infonnation  into  the  PART,  and  can  use  the  application  to  collaborate  within  the 
agency  and  with  OMB.  The  PARTWeb  application  is  also  used  to  generate  summaries  for 
ExpectMore.gov. 

Within  the  PART  schedule,  Agency  and  OMB  staff  may  find  too  little  time  remaining  to 
effectively  address  substantive  issues  if  they  wait  until  the  deadlines  to  share  initial  drafts. 
Agency  and  OMB  staffs  who  share  drafts  frequently  find  the  process  more  infonnative  and 
useful.  In  addition,  face-to-face  Agency-OMB  dialogue  in  kick-off  sessions  and/or  for  sharing 
PART  drafts  is  encouraged  for  addressing  issues  and  concerns,  and  to  coordinate  and  plan  future 
efforts. 


PROCESS  FOR  COMPLETING  THE  PART 


1.  Selecting  the  Program  and 
Determining  the  Unit  of  Analysis 

OMB  and  agencies  first  need  to  detennine  the  unit  of  analysis,  or  “program,”  to  begin  the  PART 
review.  When  choosing  programs,  agencies  may  start  by  referring  to  the  formal  budget  structure 
of  accounts  and  activities  supporting  budget  development  for  the  Executive  Branch  and  the 
Congress  and,  in  particular,  Congressional  appropriations  committees.  Although  the  budget 
structure  may  not  always  clearly  define  all  programs,  (“program  activities”  in  the  budget  are  not 
always  the  activities  that  are  managed  as  a  program  in  practice)  the  budget  structure  and 
associated  documents  are  often  comprehensive  and  readily  available  as  an  inventory  of  agency 
programs. 

A  “program”  may  also  be  a  collection  of  programs  or  activities  that  are  managed  as  one  entity  or 
that  has  a  clear  set  of  goals.  See  the  discussion  on  “Combining  Programs  for  the  PART”  for 
additional  information. 

If  OMB  and  the  agency  agree,  programs  selected  for  a  PART  review  can  be  identified  as 
aggregations  of  program  activities  in  budget  accounts,  or  in  other  meaningful  budget  breakouts 
used  for  display  to  Executive  and  Legislative  Branch  decision  makers. 

Combining  Programs  for  the  PART 

To  improve  performance  and  management  decisions,  the  PART  should  examine  the  appropriate 
unit  of  analysis.  This  is  often  the  level  where  resource  allocation  or  management  decisions  are 
made. 

As  discussed  earlier,  there  may  be  situations  in  which  it  makes  sense  to  look  at  more  than  one 
“program”  (as  recognized  in  existing  structures,  such  as  the  Budget)  within  a  single  PART.  This 
might  be  the  case  when  programs  share  a  common  goal  and  are  so  interdependent  that  it  makes 
more  sense  to  review  them  as  a  unit  than  separately. 

For  example,  several  grant  programs  within  the  Department  of  Health  and  Human  Services’ 
Administration  on  Aging  were  combined  for  consideration  under  a  single  PART  assessment. 
These  grant  programs  each  contribute  to  a  similar  program  goal  of  providing  support  to  older 
individuals  to  enable  them  to  stay  in  their  homes,  rather  than  move  to  a  long  term  care  facility. 
Often  the  same  entity  receives  multiple  grants,  and  it  is  the  combination  of  these  grants  that 
enables  individuals  to  remain  at  home.  The  data  reporting  for  these  grants  is  combined  and  the 
services  are  so  inter-related  that  assessing  the  programs  together  yield  more  meaningful 
information  for  assessing  and  improving  perfonnance. 
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Agencies  and  OMB  should  avoid,  however,  assessing  overly  aggregated  units  such  that  the 
PART  is  unable  to  illuminate  meaningful  management  distinctions  among  programs  that  share 
common  goals  or  forms  but  are  managed  differently. 

The  following  list  of  criteria  should  be  considered  in  determining  whether  to  combine  more  than 
one  program,  either  within  an  agency  or  operated  by  different  agencies,  for  PART  assessment: 

1 .  Program  Purpose.  The  proposed  combination  of  programs  should  share  a  common 
purpose  and  mission.  Issues  to  consider  include:  program  mission,  beneficiary 
characteristics,  target  populations,  grant  recipients,  etc. 

2.  Program  Design/Administration.  Programs  should  be  similarly  designed  and 
administered.  For  example  the  grantee  role,  grant  type  (e.g.  competitive,  fonnula),  benefit 
structure,  oversight  roles,  data  collection,  and/or  link  to  performance  measurement  should 
be  common  across  the  programs. 

3.  Budgeting.  The  identification  of  programs  should  relate  to  budget  decisions  because  one 
goal  of  assessing  programs  is  to  develop  information  that  will  be  useful  to  the  budget 
process.  This  does  not  necessarily  mean  the  definition  of  programs  should  be  restricted  to 
accounts  and  activities  in  the  budget  (i.e.,  a  single  budget  line),  but  rather  that  the 
programs  are  managed  as  a  single  unit.  If  programs  are  chosen  that  are  not  aligned  within 
the  budget  structure,  budget  justifications  must  include  a  cross-walk  between  the 
program(s)  and  the  budget  structure  for  public  presentation. 

4.  Performance.  The  programs  should  support  similar  long-term  outcome  goals.  The 
conclusions/recommendations  on  program  performance  should  apply  to  each  of  the 
programs  or  involve  each  one. 

In  cases  where  a  PART  encompasses  multiple  programs,  each  program  must: 

1 .  Be  addressed  by  each  question.  In  order  to  receive  a  Yes  on  a  given  question,  each  of  the 
programs  included  in  the  PART  must  receive  a  Yes  to  that  question  unless  evidence  is 
provided  on  why  the  question  is  not  relevant  to  one  of  the  programs. 

2.  Have  related  long-term  outcome  measures.  In  addition,  each  program  should  have  annual 
performance  measures.  In  the  case  of  shared  measures,  to  the  extent  possible,  each 
program  should  be  able  to  demonstrate  how  it  contributes  to  the  outcome,  output,  or 
efficiency  measured. 

Note:  More  often  programs  that  have  similar  goals  are  not  interdependent.  In  those  cases,  there 
may  be  an  advantage  to  program  comparisons  in  the  context  of  a  crosscutting  analysis  that 
reviews  the  programs’  different  administration  practices  and  differential  contributions  to  a  set  of 
outcomes.  Programs  in  a  crosscutting  analysis  should  not  be  assessed  within  a  single  PART. 


2.  Determining  the  Program  Type 

Although  most  PART  questions  are  the  same,  the  PART  divides  all  programs  into  seven 
categories  for  the  purpose  of  asking  additional  questions  unique  to  a  particular  type  of  program. 
These  categories  apply  to  both  discretionary  and  mandatory  programs. 
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1 .  Direct  Federal  Programs  Programs  where  services  are  provided  primarily  by 

employees  of  the  Federal  Government,  like  the  State 
Department’s  Visa  and  Consular  Services  program. 

Programs  that  provide  funds  to  State,  local  and  tribal 
governments,  organizations,  individuals  and  other 
entities  through  a  competitive  process,  such  as  Health 
Centers  at  the  Department  of  Health  and  Human  Services 
(HHS). 

Programs  that  provide  funds  to  State,  local  and  tribal 
governments  and  other  entities  by  formula  or  block  grant, 
such  as  the  Department  of  Energy’s  (DOE) 
Weatherization  Assistance  program  and  HHS’  Ryan 
White/ AIDS  program. 

Programs  that  accomplish  their  mission  through 
rulemaking  that  implements,  interprets  or  prescribes  law 
or  policy,  or  describes  procedure  or  practice 
requirements,  such  as  the  U.S.  Environmental  Protection 
Agency’s  Mobile  Source  Air  Pollution  Standards  and 
Certification  program. 

5.  Capital  Assets  and  Service  Programs  that  achieve  their  goals  through  development 

Acquisition  Programs  (CA)  and  acquisition  of  capital  assets  (e.g.  land,  structures, 

equipment,  and  intellectual  property)  or  the  purchase  of 
services  (e.g.  maintenance,  and  information  technology). 
Program  examples  include  Navy  Shipbuilding  and  the 
Bonneville  Power  Administration. 

6.  Credit  Programs  (CR)  Programs  that  provide  support  through  loans,  loan 

guarantees  and  direct  credit,  such  as  the  Export  Import 
Bank’s  Long  Term  Guarantees  program. 

Programs  that  focus  on  knowledge  creation  or  its 
application  to  the  creation  of  systems,  methods, 
materials,  or  technologies,  such  as  DOE’s  Solar  Energy 
and  NASA’s  Solar  System  Exploration  programs. 

There  are  PART  questions  specific  to  each  of  the  seven  types  of  Federal  programs,  though  most 
of  the  questions  are  common  across  the  seven  types.  The  vast  majority  of  Federal  programs  fit 
into  one  of  the  seven  categories  of  programs  for  which  there  is  a  PART.  Some  programs  use 
more  than  one  mechanism  to  achieve  their  goals  (e.g.,  grants  and  R&D).  If  one  type  does  not 
aptly  describe  the  functions  of  a  program,  it  may  be  appropriate  to  draw  questions  from  as  many 
as  three  different  types  to  create  a  “mixed”  assessment  and  yield  a  more  informative  assessment. 


7.  Research  and  Development 
(R&D)  Programs 


4.  Regulatory-Based  Programs 
(RG) 


3.  Block/Formula  Grant  Programs 
(BF) 


2.  Competitive  Grant  Program 
(CO) 
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In  those  instances,  the  PART  type  that  most  closely  reflects  the  core  functions  of  the  program 
should  be  chosen  as  a  primary  type  in  PARTWeb,  and  then,  if  appropriate,  questions  from  other 
PART  types  should  be  selected. 

This  document  includes  type-specific  directions  within  the  guidance  for  each  question,  as 
needed.  For  example,  R&D  programs  addressing  the  acquisition,  construction  or  operation  of 
facilities  or  other  capital  assets  should  answer  2.CA1,  3.CA1,  and  4.CA1.  On  the  other  hand, 
R&D  programs  that  use  competitive  grants,  contracts,  cooperative  agreements  or  other 
transactions  should  answer  the  Competitive  Grants  questions  (3.C01,  3.C02,  and  3.C03). 


3.  Assigning  Question  Weights 

As  a  default,  individual  questions  within  a  section  are  assigned  equal  weighting  that  total  100 
percent  for  each  section.  However,  you  may  alter  the  question  weighting  to  emphasize  key 
factors  of  the  program.  To  avoid  manipulation  of  the  total  score,  weights  must  be  adjusted 
prior  to  responding  to  any  questions.  For  example,  if  you  know  that  your  program’s  funding  is 
generally  detennined  by  policy  rather  than  performance,  you  may  decide  to  adjust  the  weight  for 
question  2.7  before  starting  the  assessment.  Significant  changes  in  question  weights  should  be 
addressed  in  the  explanations  for  the  relevant  questions. 

In  some  cases,  if  a  question  is  not  relevant  to  the  program,  you  may  deem  it  Not  Applicable.  In 
these  cases,  you  need  to  assign  a  weight  of  zero  to  the  question  and  must  provide  an  explanation 
of  this  response. 


4.  Selecting  Performance  Measures 

The  PART  contains  several  questions  about  the  quality  of  a  program’s  long-term  and  annual 
performance  measures,  as  well  as  questions  about  how  well  a  program  has  achieved  its 
performance  goals.  Because  a  program’s  performance  goals  represent  its  definition  of  success, 
the  quality  of  the  performance  goals  and  actual  perfonnance  against  those  goals  are  the  primary 
determinants  of  an  overall  PART  rating. 

The  purpose  of  this  section  of  the  guidance  is  to  help  you  define  or  select  meaningful 
performance  measures  for  programs.  Agencies  and  OMB  must  assess  the  quality  of  a 
program’s  measures:  if  they  are  meaningful  in  the  context  of  a  specific  program,  if  the 
measurement  methodology  is  sound,  and  if  the  measures  can  be  verified  with  reliable  data. 

See  “Examples  of  Performance  Measures”  at  www.omb.gov/part/. 

The  key  to  assessing  program  effectiveness  is  measuring  the  right  things,  not  just  aspects  for 
which  there  are  data.  Performance  measures  must  reflect  a  program’s  mission  and  priorities,  be 
few  in  number,  and  provide  information  to  inform  resource  allocations  and  management 
decisions.  Measures  should  reflect  desired  outcomes.  In  some  cases  where  clear  outcome 
measures  are  not  available,  comprehensive,  or  of  sufficient  quality,  it  may  be  acceptable  to  use 
output  measures,  interim  milestone  outcomes,  or  proxy  outcome  measures  to  judge  the 
program’s  success  in  achieving  desired  outcomes.  In  these  cases  it  is  important  to  provide 
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clear  justification  and  rationale  for  why  the  measures  chosen  are  appropriate,  and  provide 
comprehensive  and  quality  measurement.  In  addition,  non-experts  should  be  able  to 
understand  what  is  being  measured  and  why  it  is  relevant  to  the  program. 

Because  of  the  importance  of  perfonnance  measures,  OMB  and  agencies  must  agree  on 
appropriate  measures  early  to  allow  for  review  with  relevant  stakeholders,  if  needed.  If  the 
agency  intends  to  revise  its  strategic  goals  as  the  result  of  a  PART  assessment,  the  Government 
Performance  and  Results  Act  (GPRA)  requires  that  relevant  stakeholders  be  consulted  during  the 
strategic  plan  review. 

Basic  Concepts 

The  terms  below  will  be  used  repeatedly  in  this  guidance.  Although  further  explained  throughout 
the  document,  they  are  defined  here  as  an  introduction. 

•  Strategic  Goal  or  Strategic  Objective:  A  statement  of  aim  or  purpose  included  in  a 
strategic  plan  (required  under  GPRA).  In  a  performance  budget/performance  plan, 
strategic  goals  should  be  used  to  group  multiple  program  outcome  goals.  Each  program 
outcome  goal  should  relate  to  and  in  the  aggregate  be  sufficient  to  influence  the  strategic 
goals  or  objectives  and  their  performance  measures. 

•  Performance  Goal:  Sets  a  target  level  of  performance  over  time  expressed  as  a  tangible, 
measurable  objective,  against  which  actual  achievement  can  be  compared,  including  a 
goal  expressed  as  a  quantitative  standard,  value  or  rate.  A  performance  goal  is  comprised 
of  a  performance  measure  with  targets  and  timeframes. 

o  Performance  Measures:  Indicators,  statistics  or  metrics  used  to  gauge  program 
performance. 

o  Target:  Quantifiable  or  otherwise  measurable  characteristic  that  tells  how  well  a 
program  must  accomplish  a  perfonnance  measure. 

Categories  of  Performance  Measures 

The  PART  emphasizes  the  relationship  between  outcome,  output,  and  efficiency  measures, 
because  each  kind  of  measure  provides  valuable  information  about  program  performance. 
Collectively,  these  measures  convey  a  comprehensive  story  regarding  what  products  and  services 
agencies  provide,  how  well  they  do  so,  and  with  what  result. 

Outcome  Measures.  Outcomes  describe  the  intended  result  of  carrying  out  a  program  or  activity. 
They  define  an  event  or  condition  that  is  external  to  the  program  or  activity  and  that  is  of  direct 
importance  to  the  intended  beneficiaries  and/or  the  general  public.  For  example,  one  outcome 
measure  of  a  program  aimed  to  prevent  the  acquisition  and  transmission  of  HIV  infection  is  the 
number  (reduction)  of  new  HIV  infections  in  the  U.S. 

Output  Measures.  Outputs  describe  the  level  of  activity  that  will  be  provided  over  a  period  of 
time,  including  a  description  of  the  characteristics  (e.g.,  timeliness)  established  as  standards  for 
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the  activity.  Outputs  refer  to  the  internal  activities  of  a  program  (i.e.,  the  products  and  services 
delivered).  For  example,  an  output  could  be  the  percentage  of  warnings  that  occur  more  than  20 
minutes  before  a  tornado  forms. 

Outputs  v.  Outcomes.  While  performance  measures  must  distinguish  between  outcomes  and 
outputs,  there  must  be  a  reasonable  connection  between  them,  with  outputs  supporting  (i.e., 
leading  to)  outcomes  in  a  logical  fashion. 

Outcome  measures  are  the  most  informative  measures  about  performance,  because  they  are  the 
ultimate  results  of  a  program  that  benefit  the  public.  Programs  must  try  to  translate  existing 
measures  that  focus  on  outputs  into  outcome  measures  by  focusing  on  the  ultimate  goal  of  the 
program,  as  shown  here: 


Outputs 

Outcomes 

Number  of  housing  units  rehabilitated. 

Increases  in  equity  (property  value)  of 
rehabilitated  houses  for  low-income 
families  as  a  result  of  targeted  assistance. 

Number  of  businesses  assisted  through 
loans  and  training. 

Percent  of  businesses  that  remain  viable 

3  years  after  assistance. 

Number  of  people  served  by 
water/sewer  projects. 

Increased  percent  of  people  with  access 
to  clean  drinking  water. 

Number  of  acres  of  agricultural  lands 
with  conservation  plans. 

Percent  improvement  in  soil  quality; 
dollars  saved  in  flood  mitigation. 

Programs  that  cannot  define  a  quantifiable  outcome  measure  -  such  as  programs  that  focus  on 
process-oriented  activities  (e.g.,  data  collection,  administrative  duties  or  survey  work)  -  may 
adopt  a  “proxy”  outcome  measure.  For  example,  the  outcomes  of  a  program  that  supplies 
forecasts  through  a  tornado  warning  system  could  be  the  number  of  lives  saved  and  property 
damage  averted.  However,  given  the  difficulty  of  measuring  those  outcomes  and  the  necessity  of 
effectively  warning  to  people  in  time  to  react,  prepare,  and  respond  to  save  lives  and  property, 
the  number  of  minutes  between  the  tornado  warning  issuance  and  appearance  of  the  tornado  is  an 
acceptable  proxy  outcome  measure.  Similarly,  a  program  whose  purpose  focuses  on  processing 
claims  or  applications  may  measure  increases  in  accuracy  and  timeliness  of  service  delivery 
without  a  reduction  in  customer  satisfaction.  In  such  cases,  the  program  must  demonstrate  a  clear 
link  between  the  proxy  measure  and  the  outcomes  (the  public  good)  the  program  is  trying  to 
achieve,  and  clearly  describe  that  link  in  the  explanation  of  the  answer. 

Efficiency  Measures.  Effective  programs  not  only  accomplish  their  outcome  perfonnance  goals, 
they  strive  to  improve  their  efficiency  by  achieving  or  accomplishing  more  benefits  for  a  given 
amount  of  resources.  The  President’s  Management  Agenda  (PMA)  Budget  and  Performance 
Integration  (BPI)  Initiative  requires  agencies  to  develop  efficiency  measures  to  achieve  Green 
status.  Efficiency  measures  reflect  the  economical  and  effective  acquisition,  utilization,  and 
management  of  resources  to  achieve  program  outcomes  or  produce  program  outputs.  Efficiency 
measures  may  also  reflect  ingenuity  in  the  improved  design,  creation,  and  delivery  of  goods  and 
services  to  the  public,  customers,  or  beneficiaries  by  capturing  the  effect  of  intended  changes 
made  to  outputs  aimed  to  reduce  costs  and/or  improve  productivity,  such  as  the  improved 
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targeting  of  beneficiaries,  redesign  of  goods  or  services  for  simplified  customer  processing, 
manufacturability,  or  delivery. 

Two  categories  of  efficiency  measures  are  defined  for  the  PART.  Unless  an  efficiency  measure 
is  directly  tied  to  a  program’s  strategic  goals,  it  is  considered  an  annual  rather  than  outcome 
measure. 

•  Outcome  efficiency  measures :  The  best  efficiency  measures  capture  improvements  in 
program  outcomes  for  a  given  level  of  resource  use.  Outcome  efficiency  measures  are 
generally  considered  the  best  type  of  efficiency  measure  for  assessing  the  program 
overall.  For  example,  a  program  that  has  an  outcome  goal  of  increasing  the  participation 
of  low-income  individuals  in  advanced  placement  educational  programs  through  federal 
payment  or  partial  payment  of  test  fees  may  have  cost  per  passage  of  an  advanced 
placement  test  per  low-income  student  participant  as  an  outcome  efficiency  measure. 

•  Output  efficiency  measures:  ft  may  be  difficult  to  express  efficiency  measures  in  terms 
of  outcomes.  In  such  cases,  acceptable  efficiency  measures  could  focus  on  how  to 
produce  a  given  output  level  with  fewer  resources.  However,  this  approach  should  not 
shift  incentives  toward  quick,  low-quality  methods  that  could  degrade  program 
effectiveness  and  desired  outcomes. 

A  useful  approach  to  identifying  efficiency  measures  is  to  calculate  the  productivity  of  an  input, 
defined  as  the  ratio  of  an  outcome  or  output  to  an  input.  As  such,  input  productivity  measures 
can  be  outcome  or  output  efficiency  measures.  For  example,  the  ratio  of  the  number  of  benefit 
checks  processed  to  the  total  dollars  spent  on  federal  and  contractor  staff  during  a  pay  period  is 
an  output  (benefit  check)  efficiency  measure  of  money  spent  on  the  labor.  Input  productivity 
measures  are  relatively  simple  to  fonnulate  and  easy  to  understand.  However,  when  they  are 
used  to  as  output  efficiency  measures  to  compare  a  program’s  performance  between  different 
time  periods,  they  are  only  valid  when  the  outputs  intended  to  be  produced  within  the  time 
periods  are  comparable.  Some  aspects  of  this  issue  are  detailed  in  Appendix  D.  Concerns  about 
the  validity  of  efficiency  measures  should  be  discussed  with  your  OMB  examiner. 

Requirements  for  efficiency  measures.  As  with  all  PART  measures,  acceptable  efficiency 
measures  must  be  meaningful  in  the  context  of  the  program,  be  based  on  a  sound  measurement 
methodology,  and  can  be  verified  with  reliable  data. 

•  Meaningful  outcome  efficiency  measures  consider  the  benefit  to  the  customer  and  serve 
as  indicators  of  the  program’s  operational  performance. 

•  In  order  to  be  valid,  comparison  of  output  efficiency  measures  between  time  periods 
should  reflect  efficient  use  of  resources  rather  than  other  changes  such  as  processing  of  a 
different  output  or  a  change  in  output  mix.  Improvement  or  degradations  in  output 
efficiency  measures  over  time  should  correspond  to  the  decrease  or  increase  of  related 
costs,  respectively,  for  the  quantity  or  volume  and  mix  of  outputs.  In  all  cases,  the  use  of 
an  input  productivity  measure  as  an  output  efficiency  measure  requires  assessment  of  the 
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comparability  of  the  kinds  of  outputs  produced  during  the  measurement  periods. 
(Appendix  D  discusses  some  issues  with  these  measures.) 

•  Efficiency  measures  that  involve  a  baseline,  standard,  or  benchmark  must  be 
accompanied  by  a  history  of  the  program’s  changes  to  the  baseline,  standard,  or 
benchmark  for  all  measurement  periods.  Without  this  history,  valid  comparisons  cannot 
be  made. 

•  Efficiency  improvement  often  involves  risk,  such  as  to  quality  (e.g.,  reductions  in 
processing  time  may  increase  error  rates),  outcomes,  and  other  factors  such  as  customer 
satisfaction.  A  program  should  assess  risks  associated  with  efficiency  improvement 
efforts  and,  if  warranted,  develop  and  implement  a  risk  management  plan  that  includes 
monitoring  the  factors  at  risk. 

•  When  a  Federal  program  combines  Federal  and  non-Federal  resources,  the  inputs  for  an 
efficiency  measure  should  include  those  from  all  sources  to  the  extent  practicable. 

•  Leveraging  program  resources  can  be  a  rational  policy  decision,  as  it  leads  to  risk  or  cost 
sharing.  However,  leveraging  is  not  an  acceptable  efficiency  measure,  because  the 
leveraging  ratio  of  non-Federal  to  Federal  dollars  represents  only  inputs.  Although 
increasing  the  amount  leveraging  in  a  program  may  stretch  Federal  program  dollars,  this 
does  not  measure  improvements  in  the  management  of  the  total  program’s  resources, 
systems,  or  outcomes.  This  guideline  does  not  rule  out  the  use  of  leveraging  as  a 
performance  measure  for  other  purposes. 

Targets  and  Baselines 

Once  measures  are  defined,  ambitious  and  achievable  targets  must  be  set,  building  off  a  reliable 
baseline. 

Baselines  are  the  starting  point  from  which  gains  are  measured  and  targets  are  set.  The  baseline 
year  shows  actual  program  performance  or  prior  condition  for  the  given  measure  in  a  specified 
prior  year. 

Targets  refer  to  improved  levels  of  performance  needed  to  achieve  the  stated  goals.  These 
targets  must  be  ambitious  (i.e.,  set  at  a  level  that  promotes  continued  improvement  given 
program  circumstances)  and  achievable  given  program  characteristics.  Each  target  must  have  a 
timeframe  (e.g.,  years  in  which  the  target  level  is  to  be  achieved).  Target  setting  should  consider 
circumstances  (e.g.,  funding  levels,  changing  legislative  constraints,  past  performance)  and 
targets  may  be  adjusted  annually  as  these  factors  change. 

In  most  instances,  targets  should  be  quantifiable.  However,  in  some  cases,  like  basic  research  and 
development,  measures  and  their  targets  may  need  to  be  qualitative  and  supported  by  peer  review 
(e.g.,  expert  panels  or  Inspectors  General)  or  other  means.  When  a  target  is  not  quantitative,  it 
must  still  be  verifiable. 
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Long-term  v.  Annual  Performance  Goals 

The  PART  also  distinguishes  between  long-term  performance  goals  and  annual  performance 
goals.  While  long-term  performance  goals  address  performance  that  is  generally  several  years  or 
more  in  the  future,  annual  performance  goals  are  stated  in  yearly  increments.  Similar  to 
determining  what  a  meaningful  performance  measure  is  for  a  program,  the  appropriate  timeframe 
for  a  long-term  performance  goal  should  also  reflect  program  characteristics. 

•  Long-term  is  defined  as  covering  a  multi-year  period  of  time.  Duration  may  vary  by 
program  but  should  consider  the  nature  of  the  program  and  be  consistent  with  the  periods 
for  strategic  goals  used  in  the  Agency  Strategic  Plan. 

•  Annual  performance  goals  are  the  measures  and  targets  affected  by  an  activity  in  a 
particular  (generally  near-term)  year. 

Long-term  and  annual  measures  should  be  linked.  Indeed  a  long-term  perfonnance  goal  could  be 
an  annual  performance  goal  in  the  future.  For  example,  a  program  may  have  a  goal  of  handling 
15  million  transactions  in  201 1  for  the  same  cost  of  handling  10  million  transactions  in  2006. 

This  type  of  goal  can  demonstrate  increased  expectations  for  cost  efficiencies  over  time. 

Alternatively,  a  long-term  performance  goal  could  reflect  the  cumulative  effect  of  annual 
activities.  This  type  of  goal  can  indicate  when  the  program’s  mission  is  accomplished  or  how  it 
should  evolve  over  time. 

For  a  more  detailed  discussion  please  see  “Performance  Measurement  Challenges  and 
Strategies”  (June  2003,  http  ://www. whitehouse.  gov/ omb/part/ challenges  strategies  .html)  as  well 
as  additional  examples  of  performance  measures  on  OMB ’s  PART  website,  www.omb.gov/part/. 


5.  Answering  the  Questions 


Sections  of  the  PART 

The  PART  is  a  series  of  questions  that  assess  different  aspects  of  program  performance.  Each 
PART  is  divided  into  four  sections  (see  below).  The  answers  to  questions  in  each  of  the  four 
sections  result  in  a  numeric  score  from  0  to  100  for  the  section.  These  numeric  scores  are  tallied 
and  translated  into  qualitative  ratings:  Effective,  Moderately  Effective,  Adequate,  and  Ineffective. 
Regardless  of  the  overall  score,  a  rating  of  Results  Not  Demonstrated  is  given  when  programs  do 
not  have  acceptable  performance  measures  or  lack  baselines  and  performance  data. 

The  PART  sections  are: 

I)  Program  Purpose  &  Design:  to  assess  whether  the  program’s  purpose  and  design  are 
clear  and  sound 

II)  Strategic  Planning:  to  assess  whether  the  program  has  valid  long-term  and  annual 
measures  and  targets 
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III)  Program  Management:  to  rate  program’s  management,  including  financial  oversight 
and  program  improvement  efforts 

IV)  Program  Results/Accountability:  to  rate  program  perfonnance  on  measures  and  targets 
reviewed  in  the  strategic  planning  section  and  through  other  evaluations 

Sections  I  through  III  are  scored  in  a  Yes/No  format.  In  Section  IV,  a  four-level  scale  (Yes,  Large 
Extent,  Small  Extent,  and  No)  pennits  answers  to  reflect  partial  achievement  of  goals  and 
evidence  of  results.  Not  Applicable(NA)  may  also  be  an  appropriate  answer  as  indicated  in  this 
guidance. 

Standards  of  a  Yes 

The  PART  holds  programs  to  high  standards.  Compliance  with  the  letter  of  the  law  is  not 
enough.  A  program  must  satisfy  all  the  requirements  of  a  question  to  earn  a  Yes.  In  addition, 
those  requirements  must  be  met  fully  and  consistently.  For  instance,  the  management  practices 
assessed  in  section  3  should  be  well  established  and  routine  to  the  program’s  operations.  The 
PART  requires  a  high  level  of  evidence  to  justify  a  Yes  response,  and  credit  for  a  question  cannot 
be  given  without  evidence.  That  evidence  should  address  every  element  of  the  question,  be 
credible,  and  current  (i.e.,  from  the  last  five  years). 

Question  Linkages 

This  section  contains  guidelines  about  how  various  questions  in  the  PART  are  linked  to  one 
another. 

Establishing  appropriate  long-term  measures  and  targets  lays  the  groundwork  both  for  annual 
measures  and  targets  and  for  assessing  program  results  relative  to  those  targets.  Because  of  the 
strong  focus  on  strategic  planning  and  performance  measurement,  certain  questions  in  Sections  II 
(Strategic  Planning)  and  Section  III  (Program  Management)  are  linked  with  questions  in  Section 
IV  (Results).  Specifically,  a  program  cannot  get  full  credit  for  meeting  performance  targets  in 
Section  IV,  if  the  relevant  questions  in  Section  II  or  III  indicate  that  the  long-term,  annual,  or 
efficiency  measures  and  targets  are  not  sound.  Section  IV  scoring  allows  for  partial  achievement 
of  performance  goals  (i.e.,  Large  Extent  and  Small  Extent). 

The  table  that  follows  lists  all  PART  questions  that  are  linked,  such  that  the  answer  to  one 
question  either  prescribes  or  limits  the  answer  to  additional  questions. 
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Required  PART  Question  Linkages 

Q2.2 

Q2.3 

Q2.4 

Q2.5 

Q4.1 

Q4.2 

Q4.3 

If 

Q2.\=No 

Must 

answer 

No 

Must  provide 
explanation  of 
how  annual 
performance 
goals 

contribute  to 
long-term 
outcomes  and 
purpose  to 
receive  a  Yes 

Must 

answer  No 
if  both 
Q2.l=No 
and 

Q2.3=Ao 

Must  answer  No 
if  adequate 
outcome  (or 
output) 

measures  are  not 

available 

If  Q2.3=W> 

Must 

answer 

No 

Must 

answer  No 

If  Q2.1=Fes 
and  Q2.2  =No 

Not  higher  than 
Small  Extent 

If  Q2.3=Fes 
and  Q2.4=A<? 

Not  higher 
than  Small 
Extent 

IfQ.3.4=Ao 

Must 

answer 

No 

To  ensure  consistency  across  programs,  all  answers  must  follow  these  linkages. 

While  other  questions  are  related,  only  those  listed  above  are  linked.  For  example,  while  2.7  and 
4.5  both  address  evaluations,  the  response  to  one  does  not  determine  that  to  the  other.  A  program 
could  get  a  No  to  2.7  because  it  does  not  use  evaluation  infonnation  to  manage  but  then  receive 
credit  in  4.5  because  evaluations  document  its  success.  Another  example  of  related  but  not  linked 
questions  are  1.3  and  4.4. 

Completing  the  Explanation  and  Evidence  Sections 

Explanations.  An  explanation  must  clearly  justify  an  answer  and  address  all  aspects  of  the 
requirements  provided  in  the  guidance  for  the  question.  Several  PART  questions  have  multiple 
elements  that  must  be  addressed  to  achieve  a  Yes  answer.  When  a  question  specifies  two  or  more 
criteria  that  are  joined  by  the  word  “and,”  the  program  must  achieve  all  of  the  criteria  in  order  to 
earn  a  Yes  answer.  On  the  other  hand,  when  a  question  proposes  different  options  for  satisfying 
the  criteria,  typically  different  conditions  joined  by  the  word  “or,”  the  program  need  only  achieve 
one  criterion  to  earn  a  Yes.  In  these  cases,  the  explanation  must  address  only  the  criteria  that  have 
been  met. 

In  addition,  the  explanation  should  first  and  foremost  support  the  answer  given.  On  occasion, 
when  responses  to  the  PARTs  try  to  present  a  balanced  presentation  of  positive  and  negative 
elements,  the  resulting  explanation  becomes  unclear.  Explanations  may  present  information  on 
both  strengths  and  weaknesses,  but  must  first  state  clearly  the  principle  justification  for  the 
answer. 
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Even  where  a  program  receives  a  No,  the  explanation  should  make  clear  which  requirements  of 
the  question  the  program  meets  and  which  it  does  not. 

Evidence.  Evidence  cited  in  the  PART  should  generally  be  from  the  last  five  years. 

In  addition,  the  description  of  evidence  should  provide  information  on  both  the  source  and  basis 
for  the  judgment  used  to  detennine  the  answer.  Discussion  of  evidence  should  summarize  the 
content  of  the  evidence  and  provide  a  context  for  the  explanation  (e.g.,  evidence  provided  for  the 
performance  measurement  questions  must  compare  actual  performance  to  baseline  performance). 

When  you  cite  a  web-based  resource  or  website,  include  the  full  URL  (http://...),  verify  that  the 
address  works,  and  be  as  specific  as  possible  about  what  you  are  referencing  (including  the  page 
number,  where  possible).  Web  addresses  are  hyperlinked  in  ExpectMore.gov,  allowing  the 
reader  to  click  on  the  link  and  review  the  evidence. 

Note:  Even  if  you  paste  links  directly  from  a  text  document,  the  links  will  not  be  active  in 
PARTWeb.  URLs  are  made  active  when  the  data  from  PART  Web  is  extracted  for 
ExpectMore.gov. 

Use  of  Not  Applicable  (NA).  NA  is  an  option  when  answering  a  limited  number  of  PART 
questions.  NA  should  be  used  sparingly  and  only  when  a  question  does  not  apply  to  a  particular 
program.  NA  should  not  be  used  to  avoid  choosing  between  Yes  and  No.  NA  should  not  be  used 
in  cases  where  insufficient  evidence  exists,  the  program  needs  to  make  improvements,  or  the 
program  meets  some,  but  not  all  of  the  criteria  of  the  question.  In  all  these  cases,  the  appropriate 
answer  is  No. 

When  NA  is  selected  in  PARTWeb,  the  explanation  must  provide  a  justification  for  its  use,  the 
weight  for  the  question  must  be  reduced  to  zero,  and  the  weights  for  other  questions  in  the 
section  must  be  adjusted  as  appropriate  so  the  section  weights  sum  to  100%.  If  you  chose  the 
auto-weight  feature,  any  NA  response  will  automatically  be  set  to  zero  weight. 

To  avoid  manipulation  of  the  total  score,  NA  should  be  selected  and  justified  at  the  beginning  of 
the  PART  assessment. 

Note:  A  customized  PART  may  be  developed  by  selecting  multiple  PART  types  (e.g., 
Competitive  Grants  and  R&D).  In  those  cases,  if  type-specific  questions  are  not  relevant  for  the 
type  of  PART  being  completed,  leave  the  answers  blank  and  the  weight  zero;  do  not  enter  NA  for 
a  response. 
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INSTRUCTIONS  FOR  ANSWERING  PART  QUESTIONS 


The  following  sections  of  the  guidance  contain  question-specific  instructions  to  help  explain  the 
purpose  of  each  question  and  general  standards  for  evaluation.  These  instructions  will  not  cover 
every  case,  and  it  is  up  to  the  user  to  bring  relevant  information  to  bear  in  answering  each 
question  that  will  contribute  to  the  program's  assessment. 


Section  I. 

Program  Purpose  and  Design 

This  section  examines  the  clarity  of  program  purpose  and  soundness  of  program  design.  It  looks 
at  factors  including  those  the  program,  agency,  or  Administration  may  not  directly  control  but 
which  are  within  their  influence,  such  as  legislation  and  market  factors.  Programs  should 
generally  be  designed  to  improve  economic  efficiency,  such  as  addressing  a  public  good  or 
externality,  or  a  distributional  objective,  such  as  assisting  low-income  families  in  the  least  costly 
or  most  effective  manner.  A  clear  understanding  of  program  purpose  is  essential  to  setting 
program  goals,  measures,  and  targets;  maintaining  focus;  and  managing  the  program.  Potential 
source  documents  and  evidence  for  answering  questions  in  this  section  include  authorizing 
legislation,  the  program  description  in  the  Catalog  of  Federal  Domestic  Assistance  (CFDA), 
agency  strategic  plans,  performance  plans/performance  budgets,  and  other  agency  reports. 

Options  for  answers  are  Yes  and  No.  Design  flaws  in  the  underlying  legislation  can  and  should  be 
considered  and  supported  by  evidence,  and  are  grounds  for  a  No. 

For  R&D  programs,  most  of  the  questions  in  this  section  help  address  program  “relevance,”  one 
of  the  three  fundamental  issues  of  the  R&D  Investment  Criteria  (see  Appendix  C). 


1.1:  Is  the  program  purpose  clear? 

Purpose:  To  determine  whether  the  program  has  a  focused 
and  well-defined  mission.  Detennining  this  purpose  is 
critical  to  determination  of  useful  performance  measures 
and  targets. 


Question  1.1:  Quick  Tips 

Answer 

Options 

Yes,  No 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  the  following: 

Y  A  clear  and  unambiguous  mission.  Considerations  can  include  whether  the  program 
purpose  can  be  stated  succinctly. 


Elements  of  No:  A  No  answer  would  be  appropriate  if  the  program  has  multiple  conflicting 
purposes. 


Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 
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Evidence/Data:  Evidence  can  include  a  statement  of  the  purpose  and  supporting  objectives 
from  the  program’s  authorizing  legislation,  program  documentation  (such  as  the  program 
description  provided  by  the  agency  to  CFDA)  or  mission  statement. 


1.2:  Does  the  program  address  a  specific  and 
existing  problem,  interest,  or  need? 

Purpose:  To  detennine  whether  the  program  addresses  a 
specific  problem,  interest,  or  need  that  can  be  clearly 
defined  and  that  currently  exists. 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  each  of  the 
following: 

Y  A  relevant  and  clearly  defined  interest,  problem  or  need  exists  that  the  program  is 
clearly  designed  to  address. 

Y  The  program  purpose  is  still  relevant  to  current  conditions  (i.e.,  that  the  problem  the 
program  was  created  to  address  still  exists). 

The  explanation  should  describe  the  problem,  interest  or  need  that  the  program  is  designed  to 
address.  Considerations  could  include,  for  example,  whether  the  program  addresses  a  specific 
market  failure. 

Elements  of  No:  A  No  should  be  given  if  there  is  no  clear  need  for  the  program. 

Programs  may  receive  a  Yes  to  question  1 . 1  and  a  No  on  question  1 .2  and  vice  versa. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 

Evidence/Data:  Evidence  should  include  relevant  documentation  and  data  that  demonstrates 
the  existence  of  the  problem,  interest,  or  need.  An  example  could  be  the  number  and  income 
levels  of  uninsured  individuals  for  a  program  that  provides  care  to  those  without  health 
insurance. 


Question  1.2:  Quick  Tips 


Answer 

Options 


Yes,  No 


1.3:  Is  the  program  designed  so  that  it  is  not  redundant  or  duplicative  of  any 
other  Federal,  State,  local  or  private  effort? 

Purpose:  To  determine  whether  the  program  is  designed  to 
fill  a  unique  role  or  whether  it  instead  unnecessarily 
duplicates  or  even  competes  with  other  Federal  or  non- 
Federal  programs. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  the  following: 

N  The  program  does  not  excessively  overlap  with 


Question  1.3:  Quick  Tips 

Answer 

Options 

Yes,  No 

Additional 

Guidance 

•  Credit 

•  R&D 
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other  Federal  or  non-Federal  efforts,  including  the  efforts  of  State  and  local 
governments  or  the  private  and  non-profit  sectors. 

A  consideration  can  include  whether  the  program  serves  a  population  not  served  by  other 
programs.  Answers  should  address  redundancy  and  duplication  at  all  levels-Federal,  State, 
local  and  private  sector  efforts.  If  there  are  no  similar  programs  at  certain  levels,  state  so  in 
the  answer. 

Similar  programs  might  be  justified  in  receiving  a  Yes  if  a  strong  case  can  be  made  that  fixed 
costs  are  low  and  competition  is  beneficial  (e.g.,  perhaps  multiple  laboratories)  or  if  more 
than  one  service  delivery  mechanism  is  appropriate  (e.g.,  block  grants  for  base  activities  and 
competitive  grants  for  demonstration  projects).  Also,  the  standard  of  evidence  to  receive  a 
Yes  should  be  higher  where  Federal  programs  overlap  with  each  other  than  where  a  Federal 
program  overlaps  with  private,  local,  or  State  programs. 

Elements  of  No:  A  No  answer  should  be  given  when  there  is  more  than  one  program  that 
addresses  the  same  problem,  interest,  or  need,  regardless  of  the  size  or  history  of  the 
respective  programs.  For  programs  that  partially  overlap  with  others,  a  No  should  be  given 
when  major  aspects  of  the  program,  such  as  its  purpose,  targeted  beneficiaries,  or 
mechanisms,  are  duplicative.  If  there  are  two  programs  that  significantly  overlap  and  one  is 
large  and  another  is  small,  both  programs  should  receive  a  No  for  this  question. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 

Examples:  Two  Federal  programs  that  address  substantially  similar  training  needs  would  face 
a  high  standard  to  receive  a  Yes  to  this  question.  In  contrast,  a  Yes  could  be  given  to  a  Federal 
program  that  addressed  indigent  medical  care  across  the  nation,  even  though  there  are  many 
local  and  private  programs  that  also  address  indigent  medical  care.  The  key  would  be 
whether  the  gaps  in  the  non-Federal  provision  are  large  enough  to  warrant  a  Federal  program 
and  whether  the  Federal  program  is  well  designed  to  mesh  with  non-Federal  efforts  and 
responsibilities. 

Additional  Guidance:  For  Credit  and  R&D  programs,  there  are  a  few  additional 
considerations. 

■=>  For  Credit  programs,  a  Yes  answer  would  require  evidence  of  market  failure,  such 
that  many  credit- worthy  members  of  the  target  population  continue,  despite 
technological  and  regulatory  advances,  to  be  unable  to  obtain  credit  from  private 
lenders  or  other  Federal,  State,  or  local  programs. 

■=>  For  R&D  programs,  some  degree  of  duplication  is  permissible,  if  it  is  well  justified 
and  coordinated.  A  Yes  answer  would  require  justification  that  the  program  provides 
value  beyond  that  of  any  similar  efforts  at  the  agency,  efforts  at  other  agencies,  or 
efforts  funded  by  State  and  local  government,  private  and  non-profit  sectors,  or  other 
counties.  Justification  first  requires  due  diligence  in  identifying  similar  past  or  present 
efforts. 
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Evidence/Data:  Evidence  should  identify  duplicative  programs  and  their  total  expenditures 
and/or  descriptions  of  efforts  supported  by  those  programs  that  address  a  similar  problem  in  a 
similar  way  as  the  program  being  evaluated.  Evidence  could  include  documented  statements 
of  programs’  missions  or  activities,  other  program  reports  or  products,  and  stakeholder 
feedback.  The  CFDA  can  be  used  as  one  resource  for  identifying  potentially  duplicative 
programs  (see  http://12.46.245.173/cfda/cfda.htmf). 

1.4:  Is  the  program  design  free  of  major  flaws  that  would  limit  the  program  ’s 

effectiveness  or  efficiency? 

Purpose:  To  determine  whether  there  are  major  design 
flaws  in  the  program  that  limit  its  efficiency  or 
effectiveness. 


Elements  of  Yes :  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  The  program  is  free  from  major  design  flaws  that 
prevent  it  from  meeting  its  defined  objectives  and 
performance  goals. 

Y  There  is  no  strong  evidence  that  another  approach 
or  mechanism  would  be  more  efficient  or  effective  to  achieve  the  intended  purpose. 

A  consideration  could  be  whether  the  government  would  get  the  same  or  better  outcome  by 
expending  fewer  total  resources  through  a  different  mechanism.  For  example,  there  may  be 
evidence  that  a  regulatory  program  to  ensure  public  safety  would  be  more  effective  than  a 
grant  program.  Analysis  should  consider  whether  the  program  structure  continues  to  make 
sense  given  changing  conditions  in  the  field  (e.g.,  changing  threat  levels  or  social 
conditions).  Other  considerations  could  include  whether  the  program  extends  its  impact  by 
leveraging  funds  and  contributions  from  other  parties. 

Elements  of  No:  Unlike  other  PART  questions  that  require  evidence  to  justify  a  Yes  answer, 
the  burden  of  proof  for  this  question  is  to  provide  evidence  to  support  a  No.  If  there  is  no 
evidence  that  a  different  approach  or  mechanism  would  be  more  effective  or  efficient  given 
the  changing  conditions  in  the  field,  then  the  program  should  get  a  Yes. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 

Additional  Guidance:  For  Credit,  Regulatory,  and  Capital  Assets  and  Service  Acquisition 
programs,  there  are  a  few  additional  considerations. 

■=>  For  Credit  programs,  an  additional  consideration  is  the  incentives  the  program 
provides  to  borrowers  and  private-sector  partners  (e.g.,  lenders  in  guarantee 
programs).  A  well-designed  program  should  have  incentive  and  monitoring 
mechanisms  to  induce  borrowers  to  repay  on  time  and  to  motivate  private-sector 


Question  1.4:  Quick  Tips 


Answer 

Options 

Yes,  No 

Additional 

Guidance 

•  Credit 

•  Regulatory 

•  Capital  Assets 
and  Service 
Acquisition 
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partners  to  fully  utilize  their  expertise  to  reduce  risk  to  the  Government  and  improve 
program  effectiveness. 

■=>  For  Regulatory  programs,  a  No  should  be  given  if  the  statute  underlying  the 
regulations  is  not  designed  to  maximize  net  benefits. 

■=>  For  Capital  Assets  and  Service  Acquisition  programs,  a  Yes  answer  requires  that, 
in  addition  to  the  general  criteria,  the  program  is  supported  by  an  adequate  capital 
asset  management  infrastructure  that  is  consistent  with  the  principles  and  techniques 
of  effective  capital  programming.  It  must  have  clear  lines  of  authority,  responsibility, 
and  accountability  for  managing  capital  assets. 

Evidence/Data:  Evidence  demonstrating  efficient  design  can  include  program  evaluations 
and  cost  effectiveness  studies  comparing  alternative  mechanisms  (e.g.,  regulations  or  grants) 
with  the  current  design  (e.g.  direct  Federal  provision).  Evidence  on  the  relative  benefits  and 
costs  of  the  activity  are  also  useful. 

Evidence  for  determining  whether  the  threshold  for  capital  programming  has  been  met 
should  include  the  documented  program-relevant  agency  or  bureau  capital  programming 
policies,  directives,  instructions,  manuals,  and  assignment  of  authorities  and  responsibilities 
to  agency  personnel  and  organizational  units. 

Note  that  if  the  program’s  only  design  flaw  relates  to  targeting  of  resources  and  the  program 
received  a  No  to  question  1 .5,  it  would  not  be  appropriate  for  it  to  receive  a  No  to  this 
question  also.  However,  if  there  is  evidence  that  the  program  has  more  than  one  major  design 
flaw  (including  targeting  of  resources),  it  should  receive  a  No  to  questions  1.4  and  1.5. 

1.5:  Is  the  program  design  effectively  targeted  so  that  resources  will  address  the 
program ’s  purpose  directly  and  will  reach  intended 
beneficiaries? 

Purpose:  To  detennine  whether  the  program  is  designed  so 
that:  1)  program  resources  are  used  in  a  manner  that 
directly  and  efficiently  supports  the  program’s  purpose;  2) 
program  resources  or  outcomes  will  reach  the  intended 
beneficiaries  efficiently;  and  3)  to  avoid  unintended 
subsidies. 

“Benefits”  can  be  the  outcomes  of  program  efforts  or  direct  assistance  such  as  grant  funding. 
“Beneficiaries”  refers  to  those  who  benefit  from  the  favorable  outcome  of  the  program. 
“Reach”  refers  to  the  distribution  of  program  benefits  (i.e.  program  outcomes,  grant  funding, 
etc.). 

Unlike  Question  1.4,  which  addresses  examination  of  alternatives  to  achieve  a  program’s 
goals,  this  question  asks  whether  program  resources  under  the  current  program  design  are 
oriented  toward  the  effective  achievement  of  the  program’s  purpose. 


Question  1.5:  Quick  Tips 
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Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  each  of  the 
following: 

Y  The  program  is  designed  in  a  manner  that  ensures  resources  are  being  used  directly 
and  effectively  to  meet  the  program’s  purpose. 

Y  The  program  is  structured  so  that  resources  or  outcomes  (benefits)  will  adequately 
reach  the  intended  beneficiaries. 

Y  The  program  can  demonstrate  that  the  right  beneficiaries  are  being  targeted. 

Y  Activities  that  would  have  occurred  without  the  program  are  not  subsidized  (or 
receive  only  warranted  levels  of  subsidies). 

Acceleration  of  activities  due  to  Federal  funding  can  be  grounds  for  a  Yes,  but  there  should 
be  evidence  that  the  acceleration  warrants  the  subsidy  or  application  of  funding.  Acceleration 
of  an  activity  that  increases  profits  for  a  business  —  that  the  firm  would  or  could  have 
undertaken  eventually  without  the  subsidy  or  application  of  funding  —  would  not  generally 
qualify  for  a  Yes,  unless  there  are  significant  external  (i.e.,  social)  benefits  from  the  activity. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 

Additional  Guidance:  For  Block  Grant,  Credit,  and  R&D  programs,  there  are  a  few  additional 
considerations. 

■=>  For  Block  Grant  and  Credit  programs,  the  assessment  should  also  consider  how 
well  funds  are  targeted  to  meet  the  program  purpose  and  whether  funds  are  protected 
against  supplantation  or  substitution.  For  Credit  programs,  it  is  important  not  to 
attract  those  borrowers  who  can  obtain  credit  without  Government  assistance,  but  to 
focus  on  the  next-most-creditworthy  borrowers  who  have  a  chance  to  meet  the 
program’s  performance  goals. 

■=>  For  R&D  programs,  a  Yes  answer  would  require  identification  of  the  program’s 
relevance  to  addressing  specific  national  needs,  agency  missions,  fields  of  science  or 
technology,  or  other  “customer”  needs.  A  customer  may  be  another  program  at  the 
same  or  another  agency,  an  interagency  initiative  or  partnership,  a  firm,  or  an 
organization  from  another  sector  or  country.  For  these  programs,  the  question  refers 
to  awardees  of  contracts,  cooperative  agreements  or  other  transactions,  as  well  as 
grants. 

Evidence/Data:  Evidence  should  support  the  existence  of  an  adequate  process  for  targeting 
resources  to  the  program’s  purpose  and/or  beneficiaries.  For  example,  a  process  for 
determining  the  proper  set  of  intended  beneficiaries  should  be  flexible  enough  to  adjust  as  the 
pool  of  possible  beneficiaries  or  market  conditions  change.  Evidence  should  also  show  that 
the  program  is  designed  to  1)  reach  the  highest  practicable  percentage  of  target  beneficiaries, 
and  2)  have  the  smallest  practicable  share  of  funds  or  other  program  benefits  going  to 
unintended  beneficiaries.  Regarding  item  1,  a  small  program  may  only  be  able  to  reach  a 
small  number  of  beneficiaries,  but  it  should  be  well  targeted  on  some  merit  basis.  On  item  2, 
programs  not  designed  to  avoid  unwarranted  shares  of  funding  or  other  program  outcomes 
going  to  beneficiaries  who  do  not  need  or  merit  the  benefits  should  receive  a  No  answer. 
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Programs  that  are  designed  in  a  way  that  is  likely  to  result  in  significant  levels  of  erroneous 
payments  should  also  receive  a  No. 
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Section  II. 

Strategic  Planning 

This  section  focuses  on  program  planning,  priority  setting,  and  resource  allocation.  Key  elements 
include  an  assessment  of  whether  the  program  has  a  limited  number  of  perfonnance  measures 
with  ambitious  and  achievable  targets  to  ensure  planning,  management,  and  budgeting  are 
strategic  and  focused.  Potential  source  documents  and  evidence  for  answering  questions  include 
strategic  planning  documents,  agency  performance  plans/performance  budgets  and  reports, 
reports  and  submissions  from  program  partners,  evaluation  plans,  and  other  program  documents. 

Options  for  answers  are  Yes  and  No.  In  some  cases,  Not  Applicable  may  also  be  an  option.  While 
it  is  recognized  that  some  programs  may  have  greater  difficulty  than  others  in  developing 
quantitative  performance  goals,  programs  must  have  meaningful  and  appropriate  methods  for 
demonstrating  results.  OMB  and  agencies  should  work  together  to  develop  approaches  for 
programs  where  it  is  difficult  to  develop  quantitative  measures,  and  where  qualitative,  expert- 
review,  or  other  measures  are  more  appropriate. 

For  R&D  programs,  most  questions  in  this  section  help  address  the  prospective  aspects  of  the 
R&D  Investment  Criteria  (see  Appendix  C). 


2.1:  Does  the  program  have  a  limited  number  of  specific  long-term  performance 
measures  that  focus  on  outcomes  and  meaningfully  reflect  the  purpose  of  the 
program? 

Purpose:  To  detennine  if  the  program  has  long-term 
performance  measures  to  guide  program  management  and 
budgeting  and  promote  results  and  accountability  that  meet 
the  requirements  of  the  PART  and  GPRA.  This  question 
seeks  to  assess  whether  the  program  measures  are  salient, 
meaningful,  and  capture  the  most  important  aspects  of 
program  purpose  and  appropriate  strategic  goals. 


Question  2.1:  Quick  Tips 
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•  R&D 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  the 
following: 

N  The  program  must  have  about  two  or  three  specific,  easily  understood  long-term 
outcome  measures  that  directly  and  meaningfully  support  the  program’s  purpose. 
“Long-term”  means  a  long  period  relative  to  the  nature  of  the  program,  likely  5-10 
years,  and  consistent  with  time  periods  for  strategic  goals  used  in  the  agency’s  GPRA 
strategic  plan. 


The  performance  measures  should  focus  on  outcomes,  although  in  some  cases  output 
measures  are  permissible.  The  measures  may  be  those  developed  to  comply  with  GPRA,  so 
long  as  they  meet  the  “Perfonnance  Measures”  section  of  this  document.  Otherwise,  OMB 
and  agencies  should  revise  the  measures  to  fully  meet  GPRA  and  PART  standards.  A  Yes 
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answer  can  also  be  given  if  OMB  and  the  agency  have  reached  agreement  on  a  limited 
number  of  long-term  measures  that  will  be  added  to  the  next  GPRA  strategic  plan  or 
performance  plan/performance  budget  if  those  measures  are  included  in  the  PART. 
Significant  changes  to  the  GPRA  strategic  plan  may  require  stakeholder  consultation. 

Output  measures  only  meet  the  standards  of  a  Yes  answer  if  the  program  can  produce  sound 
justification  for  not  adopting  outcome  measures.  Whenever  output  measures  are  proposed, 
the  program  must  clearly  show  how  such  measures  reflect  progress  toward  desired  outcomes. 
The  justification  for  not  adopting  outcome  measures  and  the  explanation  of  how  output 
measures  show  progress  toward  desired  outcomes  must  be  clearly  presented  in  the 
explanation  and/or  evidence  sections.  For  more  detailed  discussion  on  when  output  measures 
may  be  used  as  proxy  measures  for  outcomes,  please  see  “Selecting  Perfonnance  Measures” 
section  of  the  PART  guidance. 

Elements  of  No:  A  No  must  be  given  for  long-term  measures  that  do  not  directly  and 
meaningfully  relate  to  the  program’s  purpose  or  are  unnecessarily  focused  on  outputs  and 
lack  adequate  justification.  A  program  should  not  receive  a  No  for  having  too  many 
measures,  if  it  has  identified  a  few  high-priority  ones  that  represent  important  aspects  of  the 
program.  A  No  must  be  given  if  the  agency  and  OMB  have  not  reached  agreement  on 
measures  that  meet  PART  requirements. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 

Examples:  A  program  that  is  exclusively  focused  on  processing  applications  and  is  unable  to 
adequately  define  a  quantifiable  outcome  measure  may  use  output  measures  that  focus  on 
increases  in  accuracy  and/or  timeliness  of  service  delivery.  An  example  of  an  unacceptable 
long-term  measure  is  a  housing  program  that  is  proposing  using  a  measure  of  “number  of 
housing  units  constructed.”  In  this  case,  such  an  output  is  unacceptable  as  the  program  is 
expected  to  articulate  and  measure  progress  toward  achieving  outcomes  (e.g.,  increases  in 
homeownership  rates,  increases  in  housing  equity  in  low-income  communities). 


Measures  and  PARTWeb 

Performance  measures  should  be  listed  in  the  Measures  screen  in 
PARTWeb.  Only  measures  that  meet  the  standards  for  a  Yes 
should  be  entered  in  the  PART.  Please  ensure  that  the  proper 
characterization  of  measures  is  selected  (i.e.,  “outcome”  or 
“output”). 


Additional  Guidance:  For  Block  Grant  and  R&D  programs,  there  are  a  few  additional 
considerations. 

■=>  For  Block  Grant  programs  that  support  a  wide  range  of  purposes  and  allow 

grantees  to  set  their  own  program  priorities,  measures  that  address  the  extent  to  which 
grantees  meet  their  own  goals  or  effectively  target  populations  are  options  if  no  better 
measures  are  possible. 
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■=>  For  R&D  programs,  OMB  will  work  with  agencies  to  assess  existing  or  develop 
appropriate  measures.  Some  R&D  programs,  especially  in  basic  research,  may  have 
difficulty  developing  long-tenn  efficiency  measures,  due  to  the  uncertainty  of 
outcomes  and  the  years  it  takes  to  achieve  and  recognize  them. 

Evidence/Data:  Evidence  will  include  the  long-term  measures  established  for  the  program 
either  in  the  existing  agency  GPRA  documents  or  other  program  documents  or  as  agreed  to 
by  OMB  and  to  be  included  in  the  next  set  of  GPRA  documents  the  agency  prepares.  In  the 
case  of  new  measures,  if  targets  and  baselines  are  not  defined,  a  plan  for  their  development 
(i.e.,  timeline,  methods  for  data  collection,  responsible  office  and/or  staff)  must  be  agreed  to 
by  the  agency  and  OMB. 

2.2:  Does  the  program  have  ambitious  targets  and  timeframes  for  its  long-term 
measures? 

Purpose:  To  determine  if  the  program  has  challenging  but 
realistic  quantifiable  targets  and  timeframes  for  the  long- 
tenn  measures. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  The  program  must  have  specific  quantified  targets 
for  measures  evaluated  in  Question  2.1.  (Where 
targets  are  not  “quantitative,”  they  still  must  be 
verifiable,  e.g.,  through  a  clean  audit  or  outstanding 
ratings  by  an  expert  panel). 

Y  The  program  is  strongly  encouraged  to  have  clear  baselines  from  which  to  measure 
targets  and  changes  in  performance  for  outcome  measures 

Y  The  targets  are  long  tenn,  likely  5-10  years. 

Y  The  program  must  have  baselines  for  output  measures. 

Y  All  targets  and  timeframes  must  be  ambitious  -  that  is,  they  must  be  set  at  a  level  that 
promotes  continued  improvement  within  achievable  efficiencies. 

Y  Where  relevant,  the  program  must  define  an  appropriate  end  target. 

Elements  of  No:  A  No  is  appropriate  if  quantified  targets  or  timeframes  are  not  included  for 
key  measures  or  if  the  targets  or  timeframes  are  not  ambitious  or  challenging.  A  No  is  also 
appropriate  where  required  baselines  have  not  been  established. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 


Question  2.2:  Quick  Tips 

Answer 

Options 

Yes,  No 

Question 

Linkages 

If  No  on  2.1,  must 
answer  No  on  2.2. 

Additional 

Guidance 

•  R&D 

Question  Linkages 

If  the  program  received  a  No  in  Question  2.1,  the  program  must  get 
a  No  for  this  question. 
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Measures  and  PARTWeb 

Targets  must  be  listed  in  the  Measures  screen  in  PARTWeb. 


Additional  Guidance:  For  R&D  programs,  there  are  a  few  additional  considerations. 

■=>  For  R&D  programs,  a  Yes  answer  would  require  that  the  program  provides  multi¬ 
year  R&D  objectives.  Where  applicable,  programs  must  provide  schedules  with 
annual  milestones,  highlighting  any  changes  from  previous  schedules.  Program 
proposals  must  define  what  outcomes  would  represent  a  minimally  effective  program 
and  a  successful  program. 

Evidence/Data:  Evidence  will  include  targets  in  existing  agency  GPRA  documents  or  other 
program  documents  or  as  agreed  to  by  OMB  and  to  be  included  in  the  2008  GPRA 
documents. 

2.3:  Does  the  program  have  a  limited  number  of  specific  annual  performance 
measures  that  can  demonstrate  progress  toward  achieving  the  program ’s  long¬ 
term  goals? 

Purpose:  To  detennine  whether  a  limited  number  of  annual 
performance  measures  have  been  identified  that  directly 
support  the  long-tenn  goals  evaluated  in  Questions  2.1  and 
2.2.  The  measures  should  be  logically  linked  to  the  long¬ 
term  goals  in  a  manner  that  enables  them  to  demonstrate 
progress  toward  achieving  those  long-term  goals. 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  A  limited  number  of  discrete,  quantifiable,  and 
measurable  annual  performance  measures  have 
been  established  for  the  program. 

Y  Annual  performance  measures  adequately  measure 
the  program's  progress  toward  reaching  the  long- 
tenn  goals  evaluated  in  Questions  2.1  and  2.2.  The 
explanation  must  clearly  state  how  the  outcomes 
help  achieve  the  long-tenn  goals  of  the  program. 

Y  Annual  performance  measures  focus  on  outcomes.  Measures  may  focus  on  outputs  if 
the  program  can  adequately  justify  why  it  is  unable  to  define  satisfactory  quantifiable 
outcome  measures.  The  justification  for  not  adopting  outcome  measures  and  the 
explanation  of  how  output  measures  show  progress  toward  desired  outcomes  must  be 
clearly  presented  in  the  explanation  and/or  evidence  sections. 

The  annual  performance  measures  may  be  those  developed  by  the  agency  to  comply  with 
GPRA,  if  the  perfonnance  measures  meet  the  criteria  listed  above. 


Question  2.3:  Quick  Tips 
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Elements  of  No:  A  No  must  be  given  if  the  agency  and  OMB  have  not  reached  agreement  on 
measures  that  meet  PART  requirements. 

Programs  must  have  at  least  one  efficiency  measure  as  part  of  their  annual  measures. 
Credit  for  efficiency  measures  is  given  in  Question  3.4. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 


Question  Relationship 

If  the  program  received  a  No  in  Question  2. 1,  an  explanation  of 
how  annual  perfonnance  goals  contribute  to  desired  long-term 
outcomes  and  the  purpose  of  the  program  must  be  provided  to 
receive  a  Yes  for  this  question. 

Measures  and  PART  Web 

Performance  measures  should  be  listed  in  the  Measures  screen  in 
PARTWeb. 

Only  measures  that  meet  the  standards  for  a  Yes  should  be  entered 
in  the  PART.  Please  ensure  that  the  proper  characterization  of 
measures  is  selected  (i.e.,  “outcome”  or  “output”). 


Additional  Guidance:  For  Block  Grant,  R&D,  and  Capital  Assets  and  Service  Acquisition 
programs,  there  are  a  few  additional  considerations. 

■=>  For  Block  Grant  programs  that  support  a  wide  range  of  purposes  and  allow 

grantees  to  set  their  own  program  priorities,  measures  that  address  the  extent  to  which 
grantees  meet  their  own  goals  or  effectively  target  populations  are  options  if  no  other 
measures  are  possible. 

■=>  For  R&D  programs,  a  Yes  answer  would  require  that  the  program  has  annual 
performance  measures  to  track  how  the  program  could  improve  scientific 
understanding  and  its  application.  For  R&D  programs  that  have  multi-year  schedules, 
the  annual  measure  should  tie  into  the  longer  term  milestones,  as  appropriate.  Some 
basic  research  programs  may  not  be  able  to  define  meaningful  annual  outcome  or 
efficiency  measures.  In  such  cases,  these  programs  may  use  process-related  indicators 
(see  Question  3.4),  if  the  program  can  explain  how  those  processes  are  directed 
toward  the  intended  long-term  goals.  OMB  will  work  with  agencies  to  address 
appropriate  measures. 

■=>  For  Capital  Assets  and  Service  Acquisition  programs,  a  Yes  answer  requires  that, 
in  addition  to  the  general  criteria,  annual  performance  measures  include  those  that  are 
sufficient  to  track  achievement  of  the  cost,  schedule,  and  performance  goals  of  asset 
acquisitions  as  they  relate  to  the  overall  program. 
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Evidence/Data:  Evidence  will  include  the  annual  measures  established  for  the  program  in  the 
agency  GPRA  performance  plan/performance  budget  or  other  program  documents,  or  they 
may  be  new  measures  as  agreed  to  by  OMB  and  which  will  be  included  in  the  2006  GPRA 
performance  plan/performance  budget.  Evidence  for  capital  asset  acquisition  programs 
includes  agency  acquisition  and  project  management  working  documents,  contract 
performance  measures  and  metrics,  and  business  cases  (OMB  Circular  A-l  1  Exhibit  300s). 

2.4:  Does  the  program  have  baselines  and  ambitious  targets  for  its  annual 
measures? 

Purpose:  To  detennine  if  the  program  has  baselines  and 
challenging,  but  realistic,  quantified  targets  for  the  annual 
measures. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  Baselines  have  been  established  for  most  of  the 
annual  measures  evaluated  in  Question  2.3. 

Y  Specific  annual  targets,  in  almost  all  cases  quantified,  have  been  developed  for  most 
of  the  program’s  annual  measures  evaluated  in  Question  2.3.  These  targets  provide  a 
specific  value  with  which  performance  can  be  compared. 

Y  At  a  minimum,  targets  are  included  for  two  years  beyond  the  one  in  which  the  PART 
was  completed. 

Y  All  targets  are  ambitious  -  that  is  they  must  be  set  at  levels  that  ensure  continued 
improvement  and  realization  of  efficiencies.  They  also  should  be  within  reason  for 
the  program  to  achieve. 

Elements  of  No:  A  No  answer  would  be  appropriate  if  quantified  targets  or  timeframes  are 
not  included  for  most  measures  or  if  the  targets  are  not  ambitious  or  challenging. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 


Question  2.4:  Quick  Tips 

Answer 

Options 

Yes,  No 

Question 

Linkages 

If  No  on  2.3,  must 
answer  No  on  2.4. 

Question  Linkages 

If  the  program  received  a  No  in  Question  2.3,  the  program  must  get 
a  No  for  this  question. 

Measures  and  PARTWeb 

Targets  should  be  listed  in  the  Measures  screen  in  PARTWeb. 


Evidence/Data:  Evidence  will  include  targets  in  the  agency  GPRA  performance 
plan/performance  budget  or  other  program  documents  or  as  agreed  to  by  OMB  and  will  be 
included  in  the  next  GPRA  performance  plan/performance  budget. 
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2.5:  Do  all  partners  (including  grantees,  sub-grantees,  contractors,  cost-sharing 
partners,  and  other  government  partners)  commit  to  and  work  toward  the  annual 
and/or  long-term  goals  of  the  program? 

Purpose:  To  determine  whether  program  efforts  carried  out 
by  program  partners  also  support  the  annual  and  long-term 
performance  goals  of  the  program. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  Partners  support  the  overall  goals  of  the  program. 

Y  Partners  measure  and  report  on  their  performance 
as  it  relates  to  accomplishing  the  program’s  goals. 

Examples:  The  most  obvious  example  of  a  partner  is  an 
entity  receiving  program  funding.  A  program  that  requires 
all  grant  agreements  and  contracts  to  include  perfonnance 
measures  that  will  help  the  program  achieve  its  goals  and  monitor  those  measures  would 
receive  a  Yes. 

While  a  program  cannot  always  control  the  activities  of  its  partners,  it  can  exert  influence 
through  a  number  of  various  mechanisms.  If  a  program  does  not  link  partners’  activities  to 
the  program’s  goals  through  a  perfonnance  requirement  or  some  other  means  then  a  No 
would  be  appropriate. 

Not  Applicable:  Not  Applicable  is  an  option  if  the  program  does  not  have  any  partners  in 
carrying  out  its  mission. 


Question  2.5:  Quick  Tips 

Answer 

Options 

Yes,  No,  NA 

Question 

Linkages 

If  No  on  2. 1  and  No 
on  2.3,  must  answer 
No  on  2.5. 

Additional 

Guidance 

•  Regulatory 

•  Programs 
funding  grants 
for  basic 
research 

Question  Linkages 

If  the  program  received  a  No  for  both  Questions  2.1  and  2.3,  the 
program  must  receive  a  No  for  this  question. 


Additional  Guidance:  For  Regulatory  programs  and  programs  funding  grants  for  basic 
research,  there  are  a  few  additional  considerations. 

■=>  For  Regulatory  programs,  all  regulated  entities  are  not  necessarily  defined  as 
program  partners. 

■=>  For  programs  funding  grants  for  basic  research,  a  Yes  answer  can  be  achieved  if 
the  program  solicitation  explicitly  includes  the  program  goals,  and  grant  applications 
and  progress  reports  provide  sufficient  means  for  the  program  manager  to  assess 
performance  and  continuing  relevance. 
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Evidence/Data:  Evidence  can  include  contracts  and  other  documents  that  tie  contractor 
performance  to  program  goals,  as  well  as  other  procedures  the  program  uses  to  get  partners  to 
commit  to,  measure,  and  report  on  performance  related  to  the  program’s  goals. 

2. 6:  Are  independent  evaluations  of  sufficient  scope  and  quality  conducted  on  a 
regular  basis  or  as  needed  to  support  program  improvements  and  evaluate 
effectiveness  and  relevance  to  the  problem,  interest, 
or  need? 

Purpose:  To  ensure  that  the  program  or  agency  conducts 
non-biased  evaluations  on  a  regular  or  as-needed  basis  to 
fill  gaps  in  perfonnance  information.  These  evaluations 
should  be  of  sufficient  scope  to  improve  planning  with 
respect  to  the  effectiveness  of  the  program. 


For  R&D  programs,  this  question  is  central  to  prospective 
planning  to  address  all  of  the  R&D  investment  criteria  (see  Appendix  C). 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  the 
following: 

Y  A  program  must  have  evaluations  that  meet  all  the  following  criteria,  described  in 
detail  below: 

■  high  quality 

■  sufficient  scope 

■  unbiased,  independent 

■  conducted  on  a  regular  basis  to  support  program  improvements 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  questions.  Given  the  flexibility  in 
determining  what  constitutes  an  evaluation,  all  programs  should  undergo  an  evaluation  that 
meets  the  following  elements  of  quality,  scope,  and  independence. 

Quality.  Evaluations  should  be  sufficiently  rigorous  to  provide  information  on  the 
effectiveness  of  the  program.  For  programs  that  support  or  employ  a  range  of  services  and 
approaches,  evaluations  should  also  provide  information  on  the  effectiveness  of  the  various 
services  and  approaches.  To  receive  a  Yes,  agencies  must  demonstrate  that  they  have  chosen 
and  applied  evaluation  methods  that  provide  the  most  rigorous  evidence  of  a  program's 
effectiveness  that  is  appropriate  and  feasible.  A  program  may  satisfy  this  criterion  if  the 
agency  and  OMB  determine  that  the  program  is  in  the  process  of  developing  new  evaluation 
approaches  that  will  provide  the  most  rigorous  evidence  possible  by  a  specified  future  date. 

The  most  significant  aspect  of  program  effectiveness  is  impact — the  outcome  of  the  program, 
which  otherwise  would  not  have  occurred  without  the  program  intervention.  A  number  of 
evaluation  methodologies  are  available  to  measure  the  effectiveness  of  programs.  Some,  such 
as  randomized  controlled  trials,  are  particularly  well  suited  to  measuring  impacts.  However, 
these  studies  are  not  suitable  or  feasible  for  every  program,  and  a  variety  of  evaluation 
methods  may  need  to  be  considered.  Other  types  of  evaluations,  including  well-designed 
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quasi-experimental  studies,  may  provide  useful  information  about  the  impact  of  a  program 
and/or  can  help  address  how  or  why  a  program  is  effective  (or  ineffective)  and  should  be 
considered  when  detennining  a  Yes  or  No  answer.  Quasi-experimental  studies  should  be 
scrutinized  given  the  increased  possibility  of  an  erroneous  conclusion. 

Overall,  evaluations  must  be  appropriate  to  the  type  and  size  of  the  program.  Agencies  and 
OMB  should  consult  evaluation  experts,  in-house  and/or  external,  as  appropriate,  when 
choosing  or  vetting  rigorous  evaluations. 

In  order  to  receive  a  Yes,  a  brief  description  of  the  nature  of  the  evaluation,  including  the 
methodology  used  and  why  it  is  sufficiently  rigorous  to  provide  evidence  of  the  effectiveness 
of  the  program  must  be  provided. 

A  list  of  additional  resources  on  evaluations  can  be  found  in  Appendix  B  of  this  document  or 
online  at  http://www.whitehouse.gov/omb/part/2004  program  eval.pdf. 

Scope.  Evaluations  must  be  sufficiently  comprehensive  to  provide  information  on  the 
effectiveness  of  the  entire  program  rather  than  just  certain  aspects  of  the  program  or  selected 
site  implementations.  In  cases  where  a  comprehensive  evaluation  is  unnecessary  based  on  the 
known  and  recently-demonstrated  effectiveness  of  a  program,  evaluations  that  fill  in  gaps  in 
the  knowledge  about  a  program’s  effectiveness  can  meet  the  elements  of  a  Yes  answer.  In  the 
absence  of  a  single  evaluation  of  sufficient  scope,  a  group  of  evaluations  may  be  used  to 
demonstrate  a  program’s  effectiveness,  provided  that  each  evaluation  meets  the  standards  for 
quality  and  independence  and  the  group  of  evaluations  together  meets  the  standard  for  scope. 
A  program  may  satisfy  this  criterion  if  the  agency  and  examiner  detennine  that  existing 
evaluations  are  inadequate,  but  the  program  is  in  the  process  of  developing  an  appropriate 
evaluation  to  be  completed  by  a  specified  future  date. 

Additional  Guidance:  For  R&D  programs,  there  are  a  few  additional  considerations. 

■=>  R&D  programs  also  should  undergo  independent  reviews  of  relevance  to  their 
agencies,  fields  of  science  or  technology,  or  customers,  in  addition  to  assessing 
questions  of  performance.  These  reviews  should  conclude  with  reports  documenting 
the  findings  and  recommendations.  A  “customer”  may  be  another  program  at  the 
same  or  another  agency,  an  interagency  initiative  or  partnership,  a  firm,  an 
organization  from  another  sector  or  country,  or  the  general  public.  Industry-relevant 
programs  may  be  expected  to  use  industry  cost-sharing  of  associated  projects  as  an 
indicator  of  market-relevance,  and  they  should  incorporate  industry  in  planning  and 
prioritization.  Reviews  should  be  rigorous  and  methodical  and  be  a  critique  of  the 
program’s  methods,  results  and  findings  by  others  in  the  field  with  requisite  training, 
expertise,  and  independence. 

Independence.  To  be  independent,  non-biased  parties  with  no  conflict  of  interest  should 
conduct  the  evaluation.  Evaluations  conducted  by  the  program  itself  should  generally  not  be 
considered  “independent;”  however,  if  the  agency  or  program  has  contracted  out  the 
evaluation  to  a  third  party  this  may  qualify  as  being  sufficiently  independent.  Evaluations 


-31  - 


conducted  by  an  agency’s  Inspector  General  or  program-evaluation  office  would  be 
considered  independent.  OMB  examiners  and  agency  staff  will  detennine  if  a  specific 
evaluation  can  be  considered  independent  for  this  question. 

The  explanation  must  describe  how  the  party  conducting  the  evaluation  is  unbiased  and  has 
no  conflict  of  interest  with  the  program  for  the  program  to  receive  a  Yes. 

Frequency.  Without  regular  updates,  program  evaluations  become  less  relevant  as  the 
information  on  which  they  were  based  becomes  older.  As  the  data  become  older,  the 
evaluations  are  less  likely  to  be  able  to  support  program  improvements.  In  order  to  capture  a 
program’s  impact  over  time,  evaluations  must  be  conducted  on  a  recurring  basis.  The  period 
of  time  between  evaluations  should  be  chosen  based  on  the  needs  and  resources  of  the 
specific  program,  but  all  programs  should  have  plans  to  repeat  evaluations  on  a  regular  basis 
with  a  specified  interval  between  assessments. 

A  program  may  satisfy  this  criterion  for  a  Yes  if  OMB  and  the  agency  agree  that  the  program 
is  developing  or  about  to  conduct  a  new  program  evaluation  in  the  near  future. 

A  program  must  have  evaluations  that  satisfy  all  four  criterion  described  above  to  receive  a 
Yes. 

Evidence/Data:  Evidence  should  include  a  program  evaluation  plan  or  schedule  of  program 
evaluations  and  program  documentation  describing  the  type  of  evaluation,  including  scope 
and  quality,  and  the  criteria  for  selecting  an  independent  evaluator. 

2. 7:  Are  Budget  requests  explicitly  tied  to  accomplishment  of  the  annual  and 
long-term  performance  goals,  and  are  the  resource  needs  presented  in  a 
complete  and  transparent  manner  in  the  program ’s 
budget? 

Purpose:  To  establish  whether  the  performance-planning 
and  budget-planning  processes  are  integrated  so  that  1) 
resource  allocation  decisions  reflect  desired  performance 
levels  (given  resource  constraints)  and  2)  the  effects  of 
funding  and  other  policy  changes  on  results  are  clear. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  The  program  must  have  program  budgeting  in  place  that  defines  the  relationship 
between  1)  annual  and  long-term  perfonnance  targets  and  2)  resources.  For 
mandatory  programs,  the  program  must  define  the  relationship  between  legislative 
policy  changes  and  program  performance. 

Y  The  program  must  have  an  integrated  budget  and  performance  presentation  that: 

■  Makes  clear  the  impact  of  funding,  policy,  or  legislative  decisions  on  expected 
performance;  and 
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■  Provides  evidence  that  the  requested  performance/resource  mix  will  enable  the 
program  to  achieve  its  performance  goals.  For  example,  from  the  information 
presented  one  should  be  able  to  answer  questions  such  as  “If  grant  funding  were 
increased  by  X,  this  program  would  be  able  to  achieve  Y  more  outcomes.” 

Y  The  program  must  report  all  direct  and  indirect  costs  needed  to  meet  performance 
targets,  including  applicable  agency  overhead,  retirement,  and  other  costs  that  might 
be  budgeted  elsewhere. 

The  exclusion  of  minor  amounts  of  services  provided  from  central  departmental  offices  (e.g., 
Office  of  the  Secretary)  from  program  costs  does  not  require  a  program  to  receive  a  No 
answer.  However,  a  program  that  generates  significant  costs  that  must  be  addressed  by 
another  program  should  budget  for  these  costs  or,  at  a  minimum,  provide  this  information  in 
clear  display  tables  that  display  the  full  costs  of  attaining  results. 

To  earn  a  Yes,  discretionary  programs  will  need  to  define  the  relationship  between  funding 
and  perfonnance  levels,  and  mandatory  programs  will  need  to  explain  the  relationship 
between  policy  changes  and  perfonnance  levels. 

Elements  of  No:  A  program  with  budget  planning  that  is  not  tied  to  performance  or  strategic 
planning  would  receive  a  No. 

Not  Applicable:  If  NA  is  selected  for  this  question,  the  explanation  must  address  why  all  of 
the  required  elements  do  not  apply  to  the  program. 

Examples:  An  energy  program  may  generate  significant  environmental  side  effects  that  must 
be  mitigated  by  a  separate  program.  The  energy  program  should  acknowledge  and  account 
for  those  mitigation  costs  as  part  of  the  full  cost  of  attaining  its  own  program  goals. 

Additional  Guidance:  For  Capital  Assets  and  Service  Acquisition  programs,  there  are  a  few 
additional  considerations. 

■=>  For  Capital  Assets  and  Service  Acquisition  programs,  a  Yes  answer  requires  that, 
in  addition  to  the  general  criteria,  programs  explain  the  relationship  of  asset 
acquisitions  to  overall  program  perfonnance  goals  and  would  be  able  to  identify 
impacts  of  changes  on  program  performance  (for  example,  the  effect  of  a  change  in 
the  quantity  acquired). 

Evidence/Data:  Evidence  can  include  explanation  and  documentation  of  how  the  budget 
request  directly  supports  achieving  performance  targets.  This  evidence  can  include  budget 
requests,  operating  plans,  and  other  documents  that  present  resource  requests.  Evidence 
should  describe  and  cite  budget  documents  that  clearly  indicate  the  full  costs  of  achieving 
performance  goals,  even  if  some  of  these  costs  do  not  appear  in  the  specific  account  or 
activity  line  of  the  program. 
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Also,  evidence  can  include  an  agency  program  budget  estimate  that  identifies  all  spending 
categories  in  sufficient  detail  to  demonstrate  that  all  relevant  costs  are  included  or  a  report 
that  shows  the  allocation  of  all  significant  program  overhead  costs  to  the  program. 

2.8:  Has  the  program  taken  meaningful  steps  to  correct  its  strategic  planning 
deficiencies? 

Purpose:  To  detennine  whether  the  program  is  on  track  to 
correct  any  strategic  planning  deficiencies  that  have  been 
identified. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  the  following: 

Y  The  program  has  acted  to  correct  strategic  planning 
deficiencies.  The  question  addresses  any 
deficiencies  identified  in  this  section.  Particular 
emphasis,  however,  should  be  placed  on  whether  the  program  is  working  to  adopt  a 
limited  number  of  specific,  ambitious  long-term  performance  goals  and  a  limited 
number  of  annual  performance  goals  that  demonstrate  progress  toward  achieving  the 
long-term  goals,  if  they  do  not  already  have  these  measures  or  associated  baselines, 
targets,  and  timeframes. 

Elements  of  No:  A  program  that  does  not  review  planning  efforts  or  does  not  make 
corrections  to  eliminate  identified  deficiencies  would  receive  a  No. 

Additional  Guidance:  For  Capital  Assets  and  Service  Acquisition  and  R&D  programs,  there 
are  a  few  additional  considerations. 

■=>  For  Capital  Assets  and  Service  Acquisition  programs  (and  relevant  R&D 

programs),  one  strategic  planning  deficiency  to  be  addressed  is  if  the  program  has  in 
the  past  received  a  No  to  Question  4.CA1. 

Evidence/Data:  Evidence  can  include  a  description  of  how  deficiencies  in  the  strategic 
planning  of  a  program  are  identified  and  corrected,  as  well  as  examples  of  such  changes.  The 
timetable  for  the  achievement  of  these  changes  should  also  be  addressed. 

Specific  Strategic  Planning  Questions  by  Program  Type 

2.RG1:  Are  all  regulations  issued  by  the  program/agency  necessary  to  meet  the 
stated  goals  of  the  program,  and  do  all  regulations  clearly  indicate  how  the  rules 
contribute  to  achievement  of  the  goals? 

(Regulatory) 

Purpose:  To  detennine  whether:  (1)  the  program  is  only 
issuing  those  rules  absolutely  necessary  to  achieve  the 
program  goals  and  is  not  over-regulating;  (2)  all  of  the 
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rules  necessary  to  meet  the  program  goals  have  been  issued;  and  (3)  the  regulations  clearly 
indicate  how  they  help  to  meet  the  program  goals.  While  previously  issued  regulations  do 
impact  this  question,  the  primary  focus  is  on  regulatory  planning  and  development. 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  each  of  the 
following: 

Y  Only  those  regulations  that  are  absolutely  necessary  to  accomplish  the  program 
mission  and  goals  are  promulgated  or  are  in  the  process  of  being  promulgated.  This 
determination  should  be  based  on  the  standards  laid  out  in  Executive  Order  12866  on 
the  identification  of  the  need  for  the  regulation  and  the  identification  of  the  market 
failure  (if  applicable). 

Y  The  regulatory  structure  is  such  that  the  public  would  be  able  to  understand  how  the 
regulations  fit  into  the  overall  achievement  of  the  program  goals. 

A  Yes  response  indicates  that  regulations  are  only  adopted  to  meet  statutory  requirements  or 
to  address  a  compelling  public  need,  such  as  a  material  failure  of  private  markets  or  planned 
or  in  the  process  of  being  promulgated  to  cover  regulatory  gaps  where  new  regulations  are 
required  to  accomplish  program  goals. 

Elements  of  No:  A  program  would  receive  a  No  if  it  1)  has  obvious  regulatory  gaps  or 
outdated  regulations  in  effect  and  2)  has  not  initiated  planned  actions  to  rectify  these 
problems  in  a  timely  manner  and  3)  does  not  have  procedures  in  place  to  minimize  regulatory 
burden. 

Evidence/Data:  Evidence  can  include  legislation  that  indicates  what  regulations  need  to  be 
promulgated  as  well  as  the  rules  themselves,  especially  the  preambles.  It  can  also  include 
internal  agency  guidance  indicating  a  process  is  in  place  to  ensure  rulemaking  involves  a 
clear  linkage  of  the  need  for  the  rule  to  a  stated  goal,  a  legal  requirement,  or  a  compelling 
public  need.  The  agency  should  be  able  to  clearly  articulate  this  association  in  the  preambles 
provided. 

2.CA1:  Has  the  agency /pro gram  conducted  a  recent,  meaningful,  credible 
analysis  of  alternatives  that  includes  trade-offs  between  cost,  schedule,  risk,  and 
performance  goals,  and  used  the  results  to  guide  the  resulting  activity?  (Capital 
Assets  and  Service  Acquisition) 

Purpose:  To  detennine  whether  the  agency  is  investing  in 
an  asset  or  service  that  provides  the  best  value  to  the 
government. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  The  agency/program  should  have  conducted  analyses  of  alternatives  and  used  those 
analyses  as  the  basis  for  a  sound  acquisition  strategy  for  the  program. 

Y  Each  analysis  should  include  the  baseline  assessment/status  quo,  non-material 
solutions  (e.g.,  data  compression  in  lieu  of  a  new  data  cable),  consideration  of 
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alternatives  (including  alternatives  to  capital  assets  and  benefit-cost  analysis  in 
accordance  with  OMB  Circular  A-94),  and  trade-offs  between  cost,  schedule,  and 
performance  goals. 

•S  The  program  should  be  able  to  demonstrate  that  the  analysis  is  credible  (i.e.,  reviewed 
and  validated  by  an  independent  entity  outside  the  program). 

Y  The  program  should  defend  differences  if  an  independent  entity’s  analysis  differs 
from  the  program’s  analysis. 

Evidence/Data:  Evidence  can  include  a  summary  of  the  analysis  of  alternatives,  and 
documentation  of  any  independent  reviews  of  the  analysis.  An  agency's  capital  asset  plan  or 
business  case  (Exhibit  300)  documentation  may  also  be  used  as  evidence. 

2.RD1:  If  applicable,  does  the  program  assess  and  compare  the  potential  benefits 
of  efforts  within  the  program  and  (if  relevant)  to  other  efforts  in  other  programs 
that  have  similar  goals?  (R&D) 

Purpose:  To  detennine  whether  applicable  programs  are 
evaluating  their  efforts  with  respect  to  their  relative 
potential  benefits.  Applicable  programs  include  those 
applied  R&D  programs  that  pursue  multiple  options 
toward  achieving  similar  public  benefits.  (This  question 
addresses  the  first  of  the  industry-related  R&D  criteria, 
regarding  the  articulation  of  program  benefits  (see  Appendix  C).) 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  each  of  the 
following: 

Y  The  program  (or  agency)  should  conduct  periodic  comparisons  of  the  potential 
benefits  of  its  proposals  with  alternatives. 

Y  The  program  should  be  able  to  demonstrate  that  the  analysis  is  credible  (i.e.,  reviewed 
and  validated  by  an  independent  entity  outside  the  program). 

Y  If  an  independent  entity’s  analysis  differs  from  the  program’s  analysis,  the  program 
should  defend  differences. 

Evidence/Data:  Evidence  can  include  an  analysis  comparing  proposed  approaches  with 
alternative  strategies.  OMB  will  work  with  agencies  as  needed  to  assist  in  the  content  and 
structure  of  these  assessments. 

R&D  programs  addressing  the  acquisition,  construction,  or  operation  of  facilities  or  other 
capital  assets  should  answer  2.CA1. 
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2.RD2:  Does  the  program  use  a  prioritization  process  to  guide  budget  requests 
and  funding  decisions?  (R&D) 

Purpose:  To  determine  whether  the  program  has  clear 
priorities  and  uses  them  in  budget  requests  and  funding 
decisions.  (This  question  addresses  the  R&D  “relevance” 
criterion  (see  Appendix  C).) 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  The  program  has  a  documented  process  to  identify  priorities. 

Y  The  program  has  evidence  that  it  uses  the  resulting  priorities  in  decision-making. 

Y  The  program  has  an  identified  set  of  current  priorities  among  program  goals, 
objectives,  and  activities. 

R&D  programs  are  encouraged  to  work  with  independent  advisory  bodies  to  help  prioritize 
in  ways  that  benefit  the  larger  science  and  technology  enterprise. 

Evidence/Data:  Evidence  can  include  clear  statements  of  program  priorities  in  program 
documentation  or  mission  statements,  as  well  as  documentation  of  the  priorities  identified  by 
any  qualified  independent  advisory  bodies.  Documentation  of  priorities  should  include  either 
a  subset  of  specific  program  activities  considered  to  be  priorities,  or  a  rank  ordering  of  all 
major,  discrete  program  activities.  Supporting  documents  should  also  describe  the  process 
used  and  factors  considered  in  determining  priorities. 
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Section  III. 

Program  Management 

This  section  focuses  on  a  variety  of  elements  related  to  whether  the  program  is  effectively 
managed  to  meet  program  performance  goals.  Key  areas  include  financial  oversight,  evaluation 
of  program  improvements,  performance  data  collection,  and  program  manager  accountability. 
Additionally,  specific  areas  of  importance  for  each  program  type  are  also  explored.  Potential 
source  documents  and  evidence  for  answering  questions  in  this  section  include  financial 
statements,  General  Accountability  Office  (GAO)  reports,  Inspector  General  (IG)  reports, 
performance  plans,  budget  execution  data,  IT  plans,  and  independent  program  evaluations. 

Options  for  answers  are  Yes  and  No.  In  some  cases,  Not  Applicable  may  also  be  an  option. 

For  R&D  programs,  some  of  the  questions  in  this  section  help  address  the  prospective  aspects  of 
program  “quality”  and  “performance”  of  the  R&D  Investment  Criteria,  in  addition  to  addressing 
general  program  management  issues  (see  Appendix  C). 


3.1:  Does  the  agency  regularly  collect  timely  and  credible  performance 
information,  including  information  from  key  program  partners,  and  use  it  to 
manage  the  program  and  improve  performance? 

Purpose:  To  detennine  whether  the  program  collects  data 
on  perfonnance  and  the  performance  of  its  partners  and 
uses  the  data  to  inform  program  management,  resource 
decisions,  and  program  performance. 


Elements  of  Yes :  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  The  program  regularly  collects  high-quality 
performance  data,  including  data  from  key  program 
partners,  relating  to  program  goals. 

Y  The  program  uses  that  information  to  adjust  program  priorities,  allocate  resources,  or 
take  other  appropriate  management  actions. 

Y  The  program  considers  the  perfonnance  of  the  program  partners  as  well  when 
assessing  progress  on  key  program  activities. 

Y  The  program  has  collected  the  baseline  perfonnance  data  necessary  to  set  meaningful, 
ambitious  performance  targets. 

Program  partners  are  other  agencies  or  intermediaries  responsible  for  carrying  out  different 
aspects  of  the  program  and  might  include  partner  agencies,  grant  recipients,  participating 
financial  institutions,  regulated  bodies,  and  contractors.  Timely  perfonnance  infonnation  is 
information  that  reflects  cunent  perfonnance  and  is  cunent  enough  to  be  useful  in  program 
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management.  Credible  perfonnance  information  is  information  that  is  collected  through  a 
systematic  process  with  quality  controls  to  confirm  the  validity  of  the  data. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 

Additional  Guidance:  For  Capital  Assets  and  Service  Acquisition,  Credit,  and  R&D 
programs,  there  are  a  few  additional  considerations. 

■=>  For  Capital  Assets  and  Service  Acquisition  programs,  a  consideration  is  whether 
the  program  uses  an  earned  value  management  system  or  similar  system. 

■=>  For  Credit  programs,  the  agency’s  comprehensive  and  current  database  should 
contain  loan-level  (or  at  least  cohort-level),  detailed  information  on  loan  perfonnance, 
bonower  characteristics,  and  lender  characteristics.  The  agency  should  employ 
adequate  information  technology,  software,  and  models  to  monitor  and  analyze  this 
information  to  effectively  manage  the  portfolios. 

■=>  For  R&D  programs,  some  long-term  basic  research  programs  may  not  be  able  to 
define  meaningful  annual  outcome  performance  measures,  aside  from  process 
measures.  In  such  cases,  these  programs  may  use  process-related  measures,  especially 
those  that  can  be  conceptually  linked  to  long-term  research  goals. 

Evidence/Data:  Evidence  can  include  a  description  of  how  the  agency  uses  perfonnance 
information  in  managing  the  program  and  how  frequently  reviews  occur,  as  well  as 
illustrative  examples  of  recent  management  actions  based  on  performance  infonnation. 
Evidence  can  also  include  steps  taken  by  a  program  to  enact  necessary  improvements  cited 
by  a  specific  evaluation. 

3.2:  Are  Federal  managers  and  program  partners  (including  grantees,  sub¬ 
grantees,  contractors,  cost-sharing  partners,  and  other  government  partners) 
held  accountable  for  cost,  schedule  and 
performance  results? 

Purpose:  To  detennine  whether  the  program  managers  and 
partners  are  accountable  for  achieving  program  results. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  The  program  identifies  the  managers  who  are 
responsible  for  achieving  key  program  results. 

Y  The  program  establishes  clearly  defined  or 
quantifiable  perfonnance  standards  and 
accountability  for  those  managers. 

Y  The  program  establishes  specific  performance 
standards  and  accountability  for  program  partners  when  program  partners  contribute 
to  the  achievement  of  program  goals. 
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Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 

Additional  Guidance:  For  Block/Formula  Grant,  Capital  Assets  and  Service  Acquisition, 
Credit,  and  R&D  programs,  there  are  a  few  additional  considerations. 

■=>  For  Block/Formula  Grant  programs,  elements  of  a  Yes  are  not  confined  to 

complying  with  the  law.  Elements  of  a  Yes  can  include  the  presence  of  incentives  for 
managers  and  program  partners  that  would  encourage  corrections  in  deficient 
programs.  For  block  grant  programs  which  support  a  wide  range  of  purposes  and 
allow  grantees  to  set  their  own  program  priorities,  this  question  should  be  interpreted 
as  whether  the  grantees  hold  managers  accountable  and  encourage  corrections  in 
deficient  programs. 

■=>  For  Capital  Assets  and  Service  Acquisition  programs,  a  consideration  is  whether 
contracts  include  minimum  performance  thresholds,  incentives  for  good  performance, 
or  other  mechanisms  to  increase  accountability. 

■=>  For  Credit  programs,  managers  and  private-sector  partners  should  also  be  held  to 
minimum  performance  thresholds  and  incentives  for  good  performance.  For  example, 
direct  loan  service  providers  or  guaranteed  lenders  should  be  monitored  on  a  regular 
basis  and  evaluated  at  least  annually  against  benchmark  default  rates. 

■=>  For  R&D  programs,  some  long-term  basic  research  programs  may  not  be  able  to 
define  meaningful  annual  outcome  performance  measures,  aside  from  process 
measures.  In  such  cases,  these  programs  may  use  process-related  measures,  especially 
those  that  can  be  conceptually  linked  to  long-term  research  goals. 

Evidence/Data:  Evidence  can  include  the  use  of  perfonnance  management  contracts  with 
program  managers,  or  some  other  mechanism  for  incorporating  program  perfonnance  into 
personnel  performance  evaluation  criteria  with  clearly  defined  or  quantifiable  performance 
targets.  Evidence  of  partners’  accountability  can  include  requiring  grant  and  contract  awards 
and  renewals  to  consider  past  perfonnance. 

3.3:  Are  funds  (Federal  and  partners’)  obligated  in  a  timely  manner,  spent  for 
the  intended  purpose,  and  accurately  reported? 

Purpose:  To  detennine  whether  funds  are  administered 
efficiently  and  obligated  in  accordance  with  planned 
schedules  and  spent  for  the  intended  purposes. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  Program  funds  are  obligated  consistently  with  the  overall  program  plan  and  that  a 
limited  amount  of  unobligated  funds  remain  at  the  end  of  the  year; 


Question  3.3:  Quick  Tips 

Answer 

Options 

Yes,  No 

-40- 


■S  Programs  and  partners  establish  schedules  for  obligations  that  properly  correspond  to 
the  resource  needs  of  the  program  plan; 

■S  Adequate  procedures  exist  for  reporting  actual  expenditures,  comparing  them  against 
the  intended  use,  and  taking  timely  and  appropriate  action  to  correct  single  audit 
findings  when  funds  are  not  spent  as  intended;  and 
■S  Program  awards  are  reported  promptly  and  accurately  (e.g.,  in  the  Federal  Assistance 
Award  Data  System  (FAADS),  Federal  Procurement  Data  System  -  Next  Generation 
(FPDS  -  NG),  etc.). 

Elements  of  No:  A  program  would  receive  a  No  if  it  had  significant  erroneous  payments  or 
was  in  violation  of  the  Anti-Deficiency  Act. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 

Evidence/Data:  Evidence  can  include  periodic  and  year-end  spending  reports  from  the 
program  and  its  partners.  Evidence  on  expenditures  can  include  spending  reports  that  draw 
intended  purpose  from  Congressional  Justifications,  Appropriations,  and  program  operating 
plans  and  match  them  against  actual  spending.  For  grantees,  evidence  can  include  grantee 
audit  reports  under  the  Single  Audit  Act,  including  data  captured  in  the  Federal  Audit 
Clearinghouse,  and  the  existence  of  an  established  procedure  for  reviewing  actual 
expenditures  against  budgets  in  grant  awards  or  appropriate  Federal  guidelines.  Evidence  for 
erroneous  payments  could  include  results  from  any  testing  completed  pursuant  to  the 
Improper  Payments  Information  Act  (IPIA)  of  2002. 

A  No  answer  to  this  question  may  warrant  extra  scrutiny  when  answering  Question  3.6, 
which  examines  financial  management  more  generally. 

3.4:  Does  the  program  have  procedures  (e.g.,  competitive  sourcing/cost 
comparisons,  IT  improvements ,  appropriate  incentives)  to  measure  and  achieve 
efficiencies  and  cost  effectiveness  in  program 
execution? 

Purpose:  To  detennine  whether  the  program  has  effective 
management  procedures  and  measures  in  place  to  ensure 
the  most  efficient  use  of  each  dollar  spent  on  program 
execution. 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  each  of  the 
following: 

■S  The  program  has  regular  procedures  in  place  to  achieve  efficiencies  and  cost 
effectiveness. 

■S  The  program  has  at  least  one  efficiency  measure  with  baseline  and  targets. 

There  are  several  ways  to  demonstrate  that  a  program  has  established  procedures  for 
improving  efficiency.  For  example,  a  program  that  regularly  uses  competitive  sourcing  to 
determine  the  best  value  for  the  taxpayer,  invests  in  IT  with  clear  goals  of  improving 


Question  3.4:  Quick  Tips 


Answer 

Options 


Yes,  No 


-41  - 


efficiency,  etc.,  could  receive  a  Yes.  A  de-layered  management  structure  that  empowers  front 
line  managers  and  that  has  undergone  competitive  sourcing  (if  necessary)  would  also 
contribute  to  a  Yes  answer.  For  mandatory  programs,  a  Yes  could  require  the  program  to  seek 
policies  (e.g.,  through  review  of  proposals  from  States)  that  would  reduce  unit  costs.  Also 
consider  if,  where  possible,  there  is  cross-program  and  inter-agency  coordination  on  IT  issues 
to  avoid  redundancies.  The  program  is  not  required  to  employ  all  these  strategies  to  earn  a 
Yes.  Rather,  it  should  demonstrate  that  efforts  improving  efficiency  are  an  established, 
regular  part  of  program  management. 

Efficiency  measures  reflect  the  economical  and  effective  acquisition,  utilization,  and 
management  of  resources  to  achieve  program  outcomes  or  produce  program  outputs.  They 
may  also  reflect  improved  design,  creation,  and  delivery  of  goods  and  services  to  the  public, 
customers,  or  beneficiaries  by  capturing  the  effect  of  intended  changes  made  to  outputs 
aimed  to  reduce  costs  and/or  improve  productivity,  such  as  the  improved  targeting  of 
beneficiaries,  redesign  of  goods  or  services  for  simplified  customer  processing, 
manufacturability,  or  delivery.  An  efficiency  measure’s  target  demonstrates  efforts  of 
program  management  to  improve  economy  in  the  acquisition  and  utilization  of  resources  as  it 
strives  to  achieve  long-tenn  and  annual  performance  goals. 

The  answer  to  this  question  should  describe  how  measures  are  used  to  evaluate  the  program’s 
success  in  achieving  efficiency  and  cost  effectiveness  improvements.  The  measures  used 
must  satisfy  the  guidelines  in  the  “Performance  Measures”  section  of  the  document. 

Elements  of  No:  A  No  must  be  given  if  the  agency  and  OMB  have  not  reached  agreement  on 
efficiency  measures  that  meet  PART  guidance. 


Measures  and  PART  Web 

To  receive  a  Yes  answer,  the  program  must  include  at  least  one 
efficiency  measure  with  baseline  data/estimates  and  targets  in  the 
Measures  screen  in  PART  Web. 

Only  measures  that  meet  the  standards  for  a  Yes  should  be  entered 
in  PARTWeb. 

Please  ensure  that  the  proper  characterization  of  measures  is 
selected  in  PARTWeb  (i.e.  “efficiency”).  Make  sure  to  indicate  the 
tenn  of  the  measure  in  PARTWeb  too  (i.e.,  long-term,  annual,  or 
long-term/ annual) . 


Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 

For  more  detailed  discussion  on  defining  acceptable  efficiency  measures  please  see  the 
section  called  “4.  Selecting  Perfonnance  Measure”  of  this  document . 
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Evidence/Data:  Evidence  can  include  efficiency  measures,  competitive-sourcing  plans,  IT 
improvement  plans  designed  to  produce  tangible  productivity  and  efficiency  gains,  or  IT 
business  cases  that  document  how  particular  projects  improve  efficiency. 

3.5:  Does  the  program  collaborate  and  coordinate  effectively  with  related 
programs? 

Purpose:  To  detennine  whether  a  Federal  program 
collaborates  with  other  related  program(s)  in  a  meaningful 
way. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  the  following: 

V  The  program  collaborates  with  related  Federal  programs  and,  to  the  extent  appropriate 
or  possible,  with  related  State,  local,  and  private  programs.  The  program  shows 
evidence  of  collaboration  leading  to  meaningful  actions  in  management  and  resource 
allocation.  The  existence  of  a  coordinating  council  would  not  by  itself  constitute 
meaningful  collaboration.  Meetings,  discussions  groups,  and  task  forces  are  not 
sufficient  for  a  Yes. 

This  question  applies  to  programs  that  have  interrelated,  but  separately  budgeted,  efforts.  An 
example  of  an  interrelated  Federal  program  is  the  shared  effort  of  the  Department  of  Veterans 
Affairs  and  the  Medicare  Program  to  provide  care  for  aging  veterans.  Likewise,  programs 
that  provide  services  to  the  same  population  at  different  times  should  also  coordinate,  such  as 
the  Department  of  Defense  and  Department  of  Veterans  Affairs,  who  both  provide  health 
care  to  service  members  at  different  points  in  time. 

Evidence/Data:  Evidence  of  meaningful  collaboration  could  include  joint  grant 
announcements,  planning  documents,  performance  goals,  or  referral  systems. 

3. 6:  Does  the  program  use  strong  financial 
management  practices? 

Purpose:  To  detennine  whether  the  program  uses  effective 
financial  management  practices  in  administering  program 
funds. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

V  The  program  is  free  of  material  internal  control 
weaknesses  reported  by  auditors. 

V  In  addition,  depending  on  the  particular  risks  inherent  to  the  program,  a  Yes  may 
require  meeting  some  or  all  of  the  following  criteria: 

■  The  program  has  procedures  in  place  to  ensure  that  payments  are  made  properly 
for  the  intended  purpose  to  minimize  erroneous  payments. 

■  Financial  management  systems  meet  statutory  requirements. 
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■  Financial  infonnation  is  accurate  and  timely. 

■  Integrated  financial  and  perfonnance  systems  support  day-to-day  operations. 

■  Financial  statements  receive  a  clean  audit  opinion  and  have  no  material  internal 
control  weaknesses. 

■  No  other  non-compliances  with  laws  and  regulations  related  to  financial 
management. 

If  an  agency-wide  material  weakness  exists  that  is  unrelated  to  the  program,  it  would  not 
prevent  the  program  from  receiving  a  Yes  response.  However,  if  an  agency- wide  material 
weakness  has  a  direct  relation  to  the  program  (e.g.,  a  lack  of  systems  that  support  day-to-day 
operations),  then  the  program  would  receive  a  No  for  this  question. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 

Additional  Guidance:  For  Block  Grant  and  Credit  programs,  there  are  a  few  additional 
considerations. 

■=>  For  Block  Grant  programs  which  support  a  wide  range  of  purposes  and  allow 
grantees  to  set  their  own  program  priorities,  a  Yes  answer  also  requires  that  grantees 
(e.g.,  States  and  localities)  meet  the  criteria  for  effective  financial  management 
identified  above. 

■=>  For  Credit  programs,  a  Yes  answer  would  also  require  that  the  program  consistently 
meets  the  requirements  of  the  Federal  Credit  Reform  Act  of  1990,  the  Debt  Collection 
Improvement  Act  and  applicable  guidance  under  OMB  Circular  A- 129.  For  Credit 
programs,  key  evidence  is  full  reconciliation  of  initial  subsidy  estimates,  re-estimates, 
and  financing  account  balances,  supplemented  by  reports  and  independent  evaluations 
of  the  program’s  portfolio  management  and  performance. 

Evidence/Data:  Evidence  can  include  recent  audit  reports  and  existence  of  procedures  to 
identify  the  above-listed  criteria,  such  as  the  ability  to  measure  improper  payments. 

Unlike  Question  3.3,  the  presence  of  significant  erroneous  payments  does  not  automatically 
generate  a  No  for  this  question.  Specifically,  if  an  agency  has  instituted  strong  financial 
management  controls  that  have  been  consistently  demonstrating  measurable  reductions  in 
erroneous  payments  over  time,  then  a  Yes  may  be  appropriate. 

3. 7:  Has  the  program  taken  meaningful  steps  to  address  its  management 
deficiencies? 

Purpose:  To  detennine  whether  the  program  has  developed 
a  system  of  evaluating  program  management  and 
correcting  deficiencies  in  a  timely  manner  once  they  are 
identified.  This  question  should  include,  but  is  not  limited 
to,  financial  management  or  other  Presidential 
Management  Agenda  deficiencies.  However,  the  focus  of 
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this  question  is  program-level  deficiencies,  as  opposed  to  agency-level  deficiencies  that  may 
not  directly  affect  the  program. 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  the 
following: 

Y  The  program  has  a  system  for  identifying  and  correcting  program  management 
deficiencies  and  uses  the  system  to  make  necessary  corrections  within  agreed  upon 
timeframes. 

Elements  of  No:  A  program  that  does  not  review  program  management  activities  and  make 
corrections  to  eliminate  identified  deficiencies  would  receive  a  No. 

Evidence/Data:  Evidence  can  include  a  description  of  how  deficiencies  in  the  program 
management  are  identified  and  corrected  as  well  as  examples  of  such  changes. 

Specific  Program  Management  Questions  by  Program  Type 

3.C01:  Are  grants  awarded  based  on  a  clear  competitive  process  that  includes  a 
qualified  assessment  of  merit?  (Competitive  Grants) 

Purpose:  To  detennine  whether  or  not  grant  funds  are 
distributed  according  to  a  competitive  process  so  that  the 
most  meritorious  applications  are  awarded  and  so  that  new 
applicants  of  merit  will  be  able  to  compete  fairly  with 
previous  grant  recipients. 

For  R&D  competitive  grants  programs,  this  question  is 
central  to  addressing  the  R&D  “quality”  criterion  (see 
Appendix  C). 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  each  of  the 
following: 

Y  The  program  operates  a  fair  and  open  competition.  Fair  and  open  does  not  mean  that 
data  should  not  be  protected,  such  as  to  meet  classification  needs,  to  protect  personal 
data  or  intellectual  property,  or  for  an  appropriate  span  of  time.  In  addition,  a  program 
may  impose  reasonable  qualification  standards,  such  as  meeting  status  as  an  academic 
organization,  and  still  satisfy  this  criterion. 

Y  The  overwhelming  majority  of  awards  (e.g.,  90  percent  on  average)  are  distributed 
according  to  a  competitive  process.  Elements  of  a  competitive  process  should  include 
independent  merit  review  and  ranking  of  applications.  Funding  awarded  through 
earmarks  would  generally  not  be  considered  competitive. 

Y  The  program  provides  a  reasonable  amount  of  outreach  to  encourage  the  participation 
of  new  grantees.  Considerations  can  include  whether  the  program  tends  to  provide 
grants  to  the  same  list  of  grantees  year  after  year. 

Y  Awards  may  be  renewed  and  still  considered  competitive  if  the  original  award  was 
competitively  awarded  and  renewals  only  extend  for  a  short  and  definite  period  of 
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time  as  long  as  the  possibility  of  renewal  is  announced  in  the  original  competition. 

The  criteria  for  renewal  and  the  renewal  process  should  be  clearly  articulated  and 
hold  renewal  applicants  to  a  high  standard  of  performance. 

Additional  Guidance:  For  relevant  R&D  programs,  there  are  a  few  additional  considerations. 

■=>  For  R&D  programs,  this  question  applies  not  only  to  grants  but  also  to  other 

funding  mechanisms  such  as  contracts  or  other  transactions.  Therefore,  competition, 
merit  review  processes  and  ranking  of  applications  should  also  apply  to  these 
transaction  vehicles. 

Evidence/Data:  Evidence  can  include  a  description  of  the  awards  process  for  grants  and 
renewals  (if  applicable),  percentage  of  funds  earmarked,  and  percentage  of  funds  subject  to 
peer  review.  Evidence  can  also  include  the  relative  number  of  new  awardees  per  award  cycle 
and  technical  assistance  and  outreach  efforts  of  the  agency. 

3.  C02:  Does  the  program  have  oversight  practices  that  provide  sufficient 
knowledge  of  grantee  activities?  (Competitive 
Grants) 

Purpose:  To  detennine  whether  or  not  the  program  has  an 
understanding  of  how  its  funds  are  utilized  by  grantees. 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  the  following: 

Y  The  program  must  have  sufficient  oversight 

capacity.  This  capacity  may  be  demonstrated  by  a 
program  that: 

■  Has  a  reporting  system  in  place  to  document  grantees’  use  of  funds  in  eligible 
activity  categories; 

■  Conducts  site  visits  to  a  substantial  number  of  grantees  on  a  regular  basis; 

■  Audits  grantee  perfonnance;  and 

■  Tracks  actual  expenditures  to  verify  that  funds  are  used  for  their  designated 
purpose. 

A  program  with  a  strong  relationship  to  its  grantees  and  a  high  level  of  understanding  of  what 
grantees  do  with  the  resources  allocated  to  them  would  receive  a  Yes. 

Elements  of  No:  A  program  with  a  poor  reporting  system  to  track  expenditures  by  grantees 
would  receive  a  No  rating. 

Additional  Guidance:  For  relevant  R&D  programs,  there  are  a  few  additional  considerations. 

■=>  For  R&D  programs,  the  above  standards  would  apply,  but  in  addition  to  grants  and 
grantees,  they  would  apply  to  contracts,  cooperative  agreements  and  other 
transactions  and  their  awardees. 
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Evidence/Data:  Evidence  can  include  the  reporting  structure,  oversight  techniques,  audit  or 
site  visit  schedule,  and/or  an  assessment  of  program  data  quality. 

3.  C03:  Does  the  program  collect  grantee  performance  data  on  an  annual  basis 
and  make  it  available  to  the  public  in  a  transparent  and  meaningful  manner? 
(Competitive  Grants) 

Purpose:  To  detennine  whether  or  not  the  program  has  a 
system  in  place  to  collect  and  present  publicly  information 
that  captures  the  most  important  impacts  of  program 
performance. 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  the  following: 

Y  The  program  collects,  compiles  and  disseminates 

grantee  perfonnance  information  in  an  accessible  manner,  such  as  via  a  web  site  or 
widely  available  program  reports.  Data  should  be  both  aggregated  on  a  program-wide 
level  and  disaggregated  at  the  grantee  level.  The  assessment  about  the  appropriate 
level  of  aggregation  of  results  may  depend  upon  needs  to  protect  certain  data,  such  as 
classified  data,  personal  data  or,  for  a  limited  span  of  time,  intellectual  property. 

Elements  of  No:  A  program  would  receive  a  No  if  grantee  perfonnance  infonnation  is  not 
available  to  the  public,  or  if  it  is  only  aggregated  at  a  high  level.  Similarly,  a  program  could 
receive  a  No  response  if  the  data  it  presents  are  not  related  to  the  impact  of  the  program. 

Additional  Guidance:  For  relevant  R&D  programs,  there  are  a  few  additional  considerations. 

■=>  For  R&D  programs,  the  above  standards  would  apply,  but  in  addition  to  grants  and 
grantees,  they  would  apply  to  contracts,  cooperative  agreements  and  other 
transactions  and  their  awardees. 

Evidence/Data:  Evidence  can  include  citations  of  the  types  of  data  that  are  collected  and 
disseminated  as  well  as  a  description  of  how  these  data  are  made  available.  The  evidence 
should  list  clearly  where  performance  information  can  be  found,  either  on  a  website  or  in 
print.  Provision  of  data  solely  through  the  Freedom  of  Information  Act  (FOIA)  process  is  not 
evidence  of  providing  infonnation  in  an  accessible  manner. 

3.BF1:  Does  the  program  have  oversight  practices  that  provide  sufficient 
knowledge  of  grantee  activities?  (Block/Formula 
Grant) 

Purpose:  To  detennine  whether  or  not  the  program  has  an 
understanding  of  how  its  funds  are  utilized  by  grantees. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  the  following: 
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•S  The  program  has  sufficient  oversight  capacity.  This  capacity  may  be  demonstrated  by 
a  program  that: 

■  Has  a  reporting  system  in  place  to  document  grantees’  use  of  funds  in  eligible 
activity  categories; 

■  Conducts  site  visits  to  a  substantial  number  of  grantees  on  a  regular  basis; 

■  Audits  grantee  perfonnance;  and 

■  Tracks  actual  expenditures  to  verify  that  funds  are  used  for  their  designated 
purpose. 

A  program  with  a  strong  relationship  to  its  grantees  and  a  high  level  of  understanding  of  what 
grantees  do  with  the  resources  allocated  to  them  would  receive  a  Yes. 

Elements  of  No:  A  program  with  no  reporting  system  to  track  expenditures  by  grantees 
would  receive  a  No. 

Evidence/Data:  Evidence  can  include  the  reporting  structure,  oversight  techniques,  audit  or 
site  visit  schedule,  and/or  an  assessment  of  program  data  quality. 

3.BF2:  Does  the  program  collect  grantee  performance  data  on  an  annual  basis 
and  make  it  available  to  the  public  in  a  transparent  and  meaningful  manner? 
(Block/Formula  Grant) 

Purpose:  To  detennine  whether  or  not  the  program  has  a 
system  in  place  to  collect  and  present  publicly  information 
that  captures  the  most  important  impacts  of  program 
performance. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  the  following: 

Y  The  program  collects,  compiles  and  disseminates  grantee  performance  information  in 
an  accessible  manner,  such  as  via  a  web  site  or  widely  available  program  reports. 
Data  would  be  both  aggregated  on  a  program-wide  level  and  disaggregated  at  the 
grantee  level. 

Elements  of  No:  A  program  would  receive  a  No  if  grantee  performance  infonnation  is  not 
available  to  the  public,  or  if  it  is  only  aggregated  at  a  high  level.  Similarly,  a  program  could 
receive  a  No  response  if  the  data  it  presents  are  not  related  to  the  impact  of  the  program. 

Evidence/Data:  Evidence  can  include  citations  of  the  types  of  data  that  are  collected  and 
disseminated  as  well  as  a  description  of  how  these  data  are  made  available.  Provision  of  data 
solely  through  the  FOIA  process  is  not  evidence  of  providing  information  in  an  accessible 
manner. 
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Note  for  Regulatory-Based  Programs 

Regulatory  questions  in  Section  III  should  be  answered  by  reviewing  examples  of 
the  program’s  recent  (e.g.,  within  the  last  five  years)  significant  rulemaking,  as 
defined  by  Executive  Order  12866.  If  the  program  has  not  completed  a  significant 
rulemaking  in  the  past  five  years  the  score  should  be  based  on  a  review  of  the 
relevant  aspects  of  the  program’s  rulemaking  process. 


3.RG1:  Did  the  program  seek  and  take  into  account  the  views  of  all  affected 
parties  (e.g.,  consumers;  large  and  small  businesses;  State,  local  and  tribal 
governments;  beneficiaries;  and  the  general  public)  when  developing  significant 
regulations?  (Regulatory) 

Purpose:  To  detennine  the  level  of  coordination,  during  the 
rulemaking  process,  with  parties  affected  by  the 
regulations. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  the  following: 

Y  The  program  solicits  the  opinions  of  affected  parties  on  significant  regulations  and 
thoroughly  evaluates  the  concerns  and  suggestions  raised  by  these  entities.  A  program 
must  seek  the  opinions  of  affected  parties  and  incorporate  their  suggestions  or  explain 
why  other  suggestions  were  not  incorporated  during  the  rule  making  process  in  order 
to  receive  a  Yes. 

While  the  element  of  seeking  views  is  mandated  by  law,  a  critical  component  of  Yes  should 
be  the  extent  to  which  the  program  takes  those  views  into  account.  If  the  program  drafts  its 
rules  in  a  vacuum  without  consulting  any  of  the  potentially  affected  parties  including  other 
affected  Federal  agencies,  it  would  likely  receive  a  No.  Additionally,  simply  seeking 
comments  is  insufficient  to  justify  a  Yes',  a  program  must  demonstrate  it  has  substantively 
considered  the  comments  received.  If  a  program  excessively  uses  or  fails  to  finalize  Interim 
Final  rules,  a  No  would  likely  be  appropriate.  Seeking  and  considering  input  from  State  and 
local  governments  in  accordance  with  EO  13132  on  Federalism  and  the  Unfunded  Mandates 
Reform  Act  is  especially  important. 

Evidence/Data:  Evidence  can  include  notices  seeking  public  comment  and  addressing 
comments  in  final  rules,  regulation  preambles  which  discuss  compliance  with  the  Regulatory 
Flexibility  Act  (RFA),  the  Unfunded  Mandates  Reform  Act  (UMRA)  of  1995,  the  Small 
Business  Regulatory  Enforcement  Fairness  Act  (SBREFA)  of  1996,  E.O.  13132,  and  the 
National  Environmental  Policy  Act,  and  detailed  preamble  language  discussing  how  public 
comments  were  considered  and  addressed. 
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3.RG2:  Did  the  program  prepare  adequate  regulatory  impact  analyses  if  required 
by  Executive  Order  12866,  regulatory  flexibility  analyses  if  required  by  the 
Regulatory  Flexibility  Act  and  SBREFA,  and  cost-benefit  analyses  if  required 
under  the  Unfunded  Mandates  Reform  Act;  and  did  those  analyses  comply  with 
OMB  guidelines?  (Regulatory) 

Purpose:  To  determine  whether  the  program,  in  justifying 
its  rules,  prepares  sound  analyses  (i.e.,  cost-benefit 
analysis,  cost  effectiveness  analysis,  and/or  risk 
assessment)  that  are  rigorous,  thorough,  and  based  upon 
the  best  available  data  and  consistent  with  OMB’s  analysis 
guidelines. 

Elements  of  Yes:  A  Yes  answer  should  be  primarily  based  on  whether  the  analyses  for  all 
significant  regulations  substantially  comply  with  EO  12866,  and  whether  the  Regulatory 
Impact  Analyses  (RIA)  for  all  economically  significant  regulations  also  substantially  comply 
with  OMB  Circular  A-4.  RIAs  for  economically  significant  rulemaking  (including  both 
proposed  and  final  rules)  should  generally  include  the  following  components: 

Y  A  statement  of  need  of  the  proposed  action  (including  any  market  failure); 

Y  An  examination  of  alternative  regulatory  and  non-regulatory  approaches  based  on  an 
appropriate  baseline;  and 

Y  An  analysis  of  the  incremental  benefits  and  costs  of  the  proposed  action  and 
alternatives  quantified  and  monetized  to  the  extent  feasible.  If  the  rule  primarily 
affects  health  and  safety,  an  incremental  cost  effectiveness  analysis  of  the  proposed 
action  and  alternatives  should  be  provided. 

When  a  rulemaking  is  based  on  scientific  analyses,  compliance  with  the  agency’s 
Information  Quality  Guidelines  and  the  requirements  of  the  OMB  Infonnation  Quality 
Bulletin  for  Peer  Review  should  be  met 

(http://www.whitehouse.gov/omb/fedreg/2005/01 1405  peer.pdf). 

Elements  of  No:  If  a  program's  impact  analyses  for  economically  significant  rules  fail  to 
include  the  required  components  listed  above,  a  No  response  to  this  question  would  be 
appropriate. 

For  all  regulations  that  have  the  potential  to  have  a  significant  impact  on  a  substantial 
number  of  small  entities,  an  agency  must  either  provide  an  analysis  of  impact  that  complies 
with  the  requirements  of  the  RFA,  or  certify  that  the  regulations  would  not  have  a  significant 
economic  impact  on  a  substantial  number  of  small  entities.  If  the  agency  so  certifies,  a 
factual  basis  must  be  provided. 

Evidence/Data:  Evidence  may  include  regulatory  impact  analyses,  regulatory  flexibility 
analyses,  cost  effectiveness  and  cost-benefit  analyses  for  the  program's  significant  and 
economically  significant  rules,  any  reports  or  feedback  generated  by  outside  reviewers, 
including  post-review  letters  resulting  from  OMB’s  review,  petitions  for  information 
corrections  under  the  Information  Quality  Act,  and  information  obtained  from  the  regulated 
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entities  and  other  important  stakeholders,  including  documents  provided  during  EO  12866 
meetings.  Evidence  on  whether  a  program’s  RIAs  include  all  of  the  required  components  can 
be  gathered  from  the  economic  data  provided  by  the  program  in  its  EO  12866  submissions 
for  economically  significant  rules. 

3.RG3:  Does  the  program  systematically  review  its  current  regulations  to  ensure 
consistency  among  all  regulations  in  accomplishing  program  goals? 

(Regulatory) 


Note  on  Question  3.RG3 

3.RG3  was  eliminated  and  the  distinctions  between  the  remaining  regulatory 
questions  were  clarified.  The  question  numbers  were  not  adjusted  as  to  maintain 
consistent  numbering  from  one  year  to  the  next. 


3.RG4:  Are  the  regulations  designed  to  achieve  program  goals,  to  the  extent 
practicable,  by  maximizing  the  net  benefits  of  its  regulatory  activity? 

(Regulatory) 

Purpose:  To  detennine  whether  the  program,  as  it 
promulgates  regulations,  selects  those  regulatory 
requirements  that  are  expected  to  maximize  net  benefits 
and/or  be  most  cost-effective.  (Note  that  this  question 
relates  to  the  promulgation  of  regulations,  as  opposed  to 
their  implementation.) 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  each  of  the 
following: 

Y  The  program  seeks  to  maximize  overall  net  benefits  to  the  greatest  extent  practicable. 

Y  The  program  should  be  able  to  demonstrate  how  the  selected  alternative  promulgated 
through  its  rulemaking  results  in  the  highest  overall  net  benefits  as  compared  to  other 
feasible  alternatives  the  agency  has  evaluated. 

An  important  consideration  for  this  question  is  whether,  in  promulgating  its  regulations,  for  a 
given  level  of  benefits,  the  agency  allows  alternative  methods  for  compliance,  record 
keeping,  and  reporting  to  minimize  the  cost  burden  on  regulated  entities  (including  electronic 
means).  Programs  should  be  judged  within  their  statutory  framework;  the  program  should 
maximize  net  benefits  to  the  extent  allowed  by  statute  in  order  to  get  a  Yes. 

Evidence/Data:  Evidence  can  include  regulatory  impact  analyses,  regulatory  flexibility 
analyses,  and  cost-benefit  and  cost  effectiveness  analyses  for  the  program’s  significant  and 
economically  significant  rules.  Evidence  should  include  a  comparison  of  significant 
programs  rules  with  other  alternatives  the  program  has  considered  on  the  basis  of  net 
expected  benefits  and  cost-effectiveness. 


Question  3.RG4:  Quick  Tips 


Answer 

Options 


Yes,  No,  NA 
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3.CA1:  Is  the  program  managed  by  maintaining  clearly  defined  deliverables, 
capability/performance  characteristics,  and  appropriate,  credible  cost  and 
schedule  goals?  (Capital  Assets  and  Service 
Acquisition) 

Purpose:  To  determine  whether  the  agency  has  clearly 
identified  and  defined  the  required  quality,  capability,  and 
performance  characteristics  or  objectives  expected  of  the 
end  product/result  of  the  asset  or  service  acquisition.  This 
element  is  critical  because  it  assures  that  all  parties  (government,  contractor,  etc.)  are 
working  toward  the  same  end-product  and  result. 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  each  of  the 
following: 

■Y  If  acquiring  a  capital  asset,  the  program  validates  planning  decisions  (procurement 
phase  step  III.  1  of  the  Capital  Programming  Guide , 
http://www.whitehouse.gov/omb/circulars/al  1/cpgtoc.html),  documents  the 
capabilities  or  characteristics  that  are  expected,  and  makes  management  decisions 
based  on  whether  milestones  are  being  met.  For  example,  a  weapon  system  that  has 
defined  key  performance  parameters  and  operational  requirements  would  get  a  Yes, 
one  that  is  proceeding  without  such  definition  should  receive  a  No. 

C  If  acquiring  services,  the  program  makes  adequate  use  of  performance-based  and 
fixed-priced  type  contracts. 

Elements  of  No:  A  program  that  acquires  services  through  other  than  performance-based  and 
fixed-priced  type  contracts  should  receive  a  No,  unless  there  is  a  legitimate  reason  for  not 
using  such  contracts. 

Evidence/Data:  Evidence  can  include  program  documentation  of  planning  decision  validation 
for  asset  acquisitions,  and  documentation  describing  key  performance  characteristics  and/or 
deliverables  and  demonstration  that  this  information  is  used  appropriately  in  management 
decisions. 

3.CR1:  Is  the  program  managed  on  an  ongoing  basis  to  assure  credit  quality 
remains  sound,  collections  and  disbursements  are  timely,  and  reporting 
requirements  are  fulfilled?  (Credit) 

Purpose:  To  determine  whether  the  program  agency  and  i 
partners  effectively  manage  and  report  on  the  risk  and 
financial  performance  of  the  program. 


Elements  of  Yes :  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

C  The  program  would  require  effectively  screen  the  risk  of  new  borrowers,  monitor 
borrowers  and  guaranteed  loan  lenders,  prevent  default,  and  resolve  defaulted  loans  in 
a  timely  manner. 


ts 


Question  3.CR1:  Quick  Tips 


Answer 

Options 


Yes,  No,  NA 


Question  3.CA1:  Quick  Tips 


Answer 

Options 


Yes,  No,  NA 
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Y  It  should  also  include  collection  and  analysis  of  borrower  repayments,  and  use  of  best 
practices  in  reducing  default  rates  and  maximizing  collections  and  recoveries. 

Evidence/Data:  Financial  reports  and  evaluations  documenting  favorable  and/or  improving 
loan  performance  relative  to  other  programs  or  private  lenders  serving  similar  borrowers, 
thorough  records,  and  evidence  of  close  cooperation  with,  and  oversight  of,  private-sector 
partners. 

3.  CR2:  Do  the  program ’s  credit  models  adequately  provide  reliable,  consistent, 
accurate  and  transparent  estimates  of  costs  and  the  risk  to  the  Government? 
(Credit) 

Purpose:  To  detennine  whether  the  program  uses  a  reliable 
method  for  estimating  program  costs. 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  The  agency  employs  a  rigorous  cost-estimation 
model  that  adequately  accounts  for  the  government's  risk  and  generates  dependable 
cost  estimates  for  each  cohort. 

Y  The  agency  should  utilize  state-of-the-art  methods  used  in  the  private  sector  for  the 
estimation  of  similar  risks. 

Elements  of  No:  The  answer  should  be  No  if  the  agency’s  data  are  not  comprehensive  or 
detailed  enough  to  apply  a  rigorous  model,  if  the  estimation  model  is  analytically  flawed,  or 
if  the  model  has  poorly  predicted  the  actual  cost  for  an  extended  period. 

Evidence/Data:  The  initial  result  should  be  reasonably  close  to  the  actual  result  most  of  the 
time,  although  not  always.  In  addition,  the  model  should  be  capable  of  explaining  prediction 
errors.  For  example,  a  model  that  relates  the  loan  default  rate  to  economic  growth  may  be 
considered  good  if  a  higher  default  rate  is  explained  by  slower  than  predicted  growth. 

3.RD1:  For  R&D  programs  other  than  competitive  grants  programs,  does  the 
program  allocate  funds  and  use  management  processes  that  maintain  program 
quality?  (R&D) 

Purpose:  To  detennine  whether  the  program  uses  a  clearly 
stated,  defensible  method  for  allocating  its  R&D  funding. 

This  question  is  central  to  addressing  the  R&D  “quality” 
criterion  (see  Appendix  C). 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  The  program  allocates  funding  using  a  broadly  competitive  process  based  on  merit,  or 
that  it  has  compelling  justifications  for  R&D  funding  allocated  through  other  means. 
Interpretations  of  competition  and  merit  review  should  be  consistent  with  the 


Question  3.RD1:  Quick  Tips 


Answer 

Options 


Yes,  No,  NA 


Question  3.CR1:  Quick  Tips 


Answer 

Options 


Yes,  No,  NA 
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definitions  in  Circular  A-l  1 :  . .intramural  and  extramural  research  programs  where 

funded  activities  are  competitively  awarded  following  review  for  scientific  and 
technical  merit.” 

•S  All  program  funds  allocated  through  means  other  than  unlimited  competition  must 
describe  and  document  the  processes  they  use  to  distribute  funds  to  each  type  of  R&D 
performer  (e.g.,  Federal  laboratories,  Federally-funded  R&D  centers,  universities, 
etc.). 

•S  Programs  are  encouraged  to  use  external  assessment  of  the  methods  they  use  to 
allocate  R&D  and  maintain  program  quality. 

■S  Programs  must  justify  the  unique  capabilities  of  each  project  performer  (Federal  lab, 
FFRDC,  or  other)  that  was  allocated  or  awarded  funds  through  a  non-competitive 
process. 

Evidence/Data:  Evidence  should  include  a  description  of  the  awards  process  and  results  of 
external  assessments  of  project  perfonners,  if  available.  Evidence  should  also  include  a 
description  of  the  unique  capabilities  of  project  perfonners  awarded  funds  through  non¬ 
competitive  processes.  If  the  program  allocates  funds  through  a  mix  of  competitive  and  non¬ 
competitive  processes  (including  earmarks),  then  the  explanation  or  evidence  must  include  a 
full  accounting  of  the  allocation  of  funding  for  the  most  recent  year  available,  at  a  minimum 
(e.g.,  percent  of  funds  awarded  competitively,  non-competitively,  eannarks,  etc.). 

R&D  programs  must  complete  question  3.RD1  if  they  allocate  funding  for  intramural 
research,  or  if  they  allocate  funds  to  Federally-funded  R&D  Centers  (FFRDCs)  or  other 
project  performers  without  using  a  competitive  process.  For  government-owned  contractor- 
operated  facilities  (e.g.,  many  FFRDCs),  competitive  selection  of  a  maintenance  and 
operation  contractor  does  not  constitute  allocation  of  R&D  funds  via  a  competitive  process. 

R&D  programs  that  use  only  competitive  grants  to  allocate  funds  should  not  answer  3.RD1. 
However,  R&D  programs  that  use  competitive  grants,  contracts,  cooperative  agreements  or 
other  transactions  as  well  as  intramural  and/or  noncompetitive  processes  should  answer  both 
the  Competitive  Grants  questions  (3.C01,  C02,  C03)  and  3.RD1. 

R&D  programs  addressing  the  acquisition,  construction  or  operation  of  facilities  or  other 
capital  assets  should  answer  the  Capital  Assets  and  Service  Acquisition  question  (3.CA1). 
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Section  IV. 

Program  Results/ Account  ability 

This  section  considers  whether  a  program  is  meeting  its  long-term  and  annual  performance  goals. 
This  section  also  assesses  how  well  the  program  compares  to  similar  programs  and  how  effective 
the  program  is  based  on  independent  evaluations.  Potential  source  documents  and  evidence  for 
answering  questions  in  this  section  include  GPRA  performance  reports,  evaluations,  GAO 
reports,  IG  reports  and  other  agency  documents.  Assessments  of  program  results  should  be  based 
on  the  most  recent  reporting  cycle  or  other  relevant  data.  PARTWeb  contains  data  fields  for  a 
performance  targets  and  results,  and  should  be  completed  to  the  greatest  extent  possible  for  all 
measures  agreed  to  by  OMB  and  the  agency  for  Section  II. 

Answers  in  this  section  are  rated  as  Yes,  Large  Extent,  Small  Extent,  and  No.  In  some  cases,  Not 
Applicable  may  also  be  an  option.  Like  Sections  I-III,  the  scoring  system  in  this  section  remains 
on  a  0  to  1  point  scale.  Scoring  for  this  section  differs  by  including  the  option  of  partial  credit 
between  0  and  1  in  increments  of  0,  .33,  .67,  and  1.  Explanations  should  address  the  basis  for 
determining  the  amount  of  credit  given. 

In  general,  Not  Applicable  answers  are  not  appropriate  for  Questions  4.1,  4.2,  4.3,  and  4.5.  While 
it  is  recognized  that  some  programs  may  have  great  difficulty  developing  quantitative 
performance  goals,  programs  are  strongly  encouraged  to  have  some  meaningful  and  appropriate 
methods  for  demonstrating  results.  OMB  and  agencies  should  work  together  to  develop 
approaches  for  programs  where  it  is  difficult  to  develop  quantitative  measures,  and  where 
qualitative,  expert-review,  or  other  measures  are  more  appropriate. 

For  R&D  programs,  most  of  the  questions  in  this  section  help  address  the  retrospective  aspects 
of  the  R&D  Investment  Criteria,  with  emphasis  on  the  “performance”  criterion;  see  Appendix  C. 


4.1:  Has  the  program  demonstrated  adequate  progress  in  achieving  its  long-term 
performance  goals? 

Purpose:  To  detennine  whether  the  program  is  meeting  or 
making  measurable  progress  toward  meeting  the  long-term 
performance  goals  evaluated  in  Questions  2.1  and  2.2.  The 
question  also  seeks  to  determine  whether  the  program's 
partners  are  meeting  long-term  goals  evaluated  in  Question 
2.5,  if  partner  performance  is  critical  to  the  program 
achieving  its  goals.  Examples  of  partners  can  include 
grantees,  participating  financial  institutions,  regulated 
bodies,  or  suppliers. 

Elements  of  Yes:  A  Yes  answer  (i.e.,  full  credit)  needs  to 
clearly  explain  and  provide  evidence  of  each  of  the 
following: 


Question  4.1:  Quick  Tips 

Answer 

Options 

Yes,  Large  Extent, 
Small  Extent,  No 

Question 

Linkages 

If  No  on  2.1,  No  on 
4.1. 

If  Yes  on  2.1  and 

No  on  2.2,  no 
higher  than  Small 
Extent  on  4.1. 
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•S  The  program  must  be  on  track  to  meet  all  the  long-term  perfonnance  goals  - 

including  ambitious  targets  and  timeframes  -  evaluated  in  Questions  2.1  and  2.2.  A 
program  would  not  receive  a  Yes  answer  by  simply  meeting  any  one  of  its  long-term 
targets,  or  by  having  performance  measures  but  no  ambitious  targets  and  timeframes. 

Y  Where  applicable,  partners  commit  to  long-tenn  outcome  targets  and  achieve  them  as 
well. 

Y  Where  relevant,  the  program  should  have  addressed  appropriately  any  predefined  end 
targets. 

Partial  credit,  such  as  Large  Extent  or  Small  Extent,  should  be  given  in  cases  where  there  is 
partial,  but  notable,  achievement  of  long-tenn  targets.  A  program  could  receive  a  No  if  it  had 
received  a  Yes  for  achieving  its  annual  targets  (Question  4.2),  but  is  not  making  progress 
toward  meeting  its  long-term  goals. 

Any  answer  other  than  No  requires  that  actual  measures,  targets,  and  data  be  entered  in  the 
Measures  section. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 


I  Additional  Rating  Guidance 

No  credit  can  be  given  for  progress  made  toward  measures,  goals, 
or  targets  that  are  not  approved  in  Question  2.1. 

Question  Linkages 

If  the  program  received  a  No  in  Question  2.1,  the  program  must 
receive  a  No  answer  to  this  question. 

If  the  program  received  a  Yes  in  Question  2.1  and  a  No  in  Question 
2.2,  then  the  program  cannot  receive  a  rating  higher  than  Small 
Extent. 


Evidence/Data:  To  support  a  Yes  or  Large  Extent,  the  Measures  screen  in  PART  Web  must 
include  historical  performance  data  showing  the  program’s  successful  progress  in  meeting  its 
long-term  perfonnance  goals.  Evidence  can  also  include  data  from  the  agency's  GPRA 
performance  report,  a  strategic  plan,  or  other  Administration  goals  and  objectives.  Reports 
detailing  customer  satisfaction  with  program  performance,  program  reports  detailing  rates  of 
utilization  or  participation,  or  independent  evaluations  of  the  program’s  performance  may 
also  be  considered  as  relevant  evidence.  In  cases  where  targets  are  not  met,  additional 
evidence  can  include  an  explanation  of  the  main  reasons. 


Measures  and  PART  Web 

Space  is  provided  in  the  Measures  screen  in  PARTWeb  to  list  and 
document  goals,  targets  and  achieved  results.  Only  measures  that 
meet  the  standards  for  a  Yes  should  be  entered  in  the  PART. 
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4.2:  Does  the  program  (including program  partners)  achieve  its  annual 
performance  goals? 

Purpose:  To  determine  whether  the  program  is  meeting  the 
targets  evaluated  in  Question  2.4.  The  question  also  seeks 
to  determine  whether  the  program's  partners  are  meeting 
annual  targets  evaluated  in  Question  2.5,  if  partner 
performance  is  critical  to  the  program  achieving  its  overall 
targets.  Examples  of  partners  can  include  grantees, 
contractors,  participating  financial  institutions,  regulated 
bodies,  or  suppliers. 

Elements  of  Yes:  A  Yes  answer  (i.e.,  full  credit)  needs  to 
clearly  explain  and  provide  evidence  of  each  of  the 
following: 

Y  The  program  meets  all  the  annual  perfonnance 
targets  evaluated  in  Question  2.4. 

Y  The  program  received  a  Yes  for  Questions  2.1  and  2.3,  and  a  Yes  or  Not  Applicable 
for  Question  2.5.  A  program  would  not  receive  a  Yes  answer  by  simply  meeting  any 
one  of  its  annual  targets. 

Y  Where  applicable,  partners  commit  to  annual  targets  and  achieve  them  as  well. 

Partial  credit  such  as  Large  Extent  or  Small  Extent  should  be  given  in  cases  where  there  is 
partial,  but  notable,  achievement  of  targets.  Any  answer  other  than  No  requires  that  actual 
measures,  targets,  and  data  be  entered  in  the  Measures  section. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 


Question  4.2:  Quick  Tips 

Answer 

Options 

Yes,  Large  Extent, 
Small  Extent,  No 

Question 

Linkages 

If  No  on  2.3,  No  on 
4.2. 

If  Yes  on  2.3  and 

No  on  2.4,  no 
higher  than  Small 
Extent  on  4.2. 

Additional  Rating  Guidance 

No  credit  can  be  given  for  progress  made  toward  measures,  goals, 
or  targets  that  are  not  approved  in  Question  2.3. 

Question  Linkages 

If  a  program  received  a  No  in  Question  2.3,  the  program  must 
receive  a  No  answer  to  this  question. 

If  the  program  received  a  Yes  in  Question  2.3  and  a  No  in  Question 
2.4,  then  the  program  cannot  receive  a  rating  higher  than  Small 
Extent. 


Evidence/Data:  To  support  a  Yes  or  Large  Extent,  the  Measures  section  in  PARTWeb  must 
include  historical  performance  data  showing  the  program’s  success  in  meeting  its  annual 
performance  goals.  Evidence  can  also  include  data  from  the  agency's  annual  GPRA 
performance  report,  a  strategic  plan,  or  other  Administration  goals  and  objectives.  In  cases 
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where  targets  are  not  met,  additional  evidence  can  include  an  explanation  of  the  main 
reasons. 


!  Measures  and  PARTWeb 

Space  is  provided  in  the  Measures  screen  in  PARTWeb  to  list  and 
document  goals,  targets  and  achieved  results.  Only  measures  that 
meet  the  standards  for  a  Yes  should  be  entered  in  the  PART. 


4.3:  Does  the  program  demonstrate  improved  efficiencies  or  cost  effectiveness  in 
achieving  program  goals  each  year? 

Purpose:  To  detennine  whether  management  practices 
have  resulted  in  efficiency  gains  over  the  past  year. 

Elements  of  Yes :  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  each  of  the  following: 

Y  The  program  demonstrated  improved  efficiency  or 
cost  effectiveness  over  the  prior  year,  including 
meeting  its  efficiency  target  in  Question  3.4.  When 
possible,  the  explanation  should  include  specific  information  about  the  program’s 
annual  savings  over  the  prior  year  as  well  as  what  the  program  did  to  achieve  the 
savings. 

The  results  of  efficiency  improvements  should  be  measured  in  terms  of  dollars  to  the 
maximum  extent  possible.  For  example,  programs  that  complete  an  A-76  competition  -  an 
indicator  of  cost-efficient  processes  -  would  contribute  to  a  Yes  answer,  provided  that  the 
competition  resulted  in  savings. 


Question  4.3:  Quick  Tips 

Answer 

Options 

Yes,  Large  Extent, 
Small  Extent,  No 

Question 

Linkages 

If  No  on  3.4,  No  on 
4.3. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question. 


Question  Linkages 

If  a  program  received  a  No  in  Question  3.4,  the  program  must 
receive  a  No  answer  to  this  question. 


Evidence/Data:  Evidence  can  include  meeting  efficiency  measure  targets  to  reduce  per  unit 
costs  or  meeting  other  targets  that  result  in  tangible  productivity  or  efficiency  gains. 
Efficiency  measures  may  also  be  considered  in  Questions  4.1  and  4.2. 
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4.4:  Does  the  performance  of  this  program  compare  favorably  to  other  programs, 
including  government,  private,  etc.,  with  similar 
purpose  and  goals? 

Purpose:  To  determine  how  well  the  program  performs 
relative  to  other  programs  that  work  toward  similar 
purposes  or  goals. 


Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain 
and  provide  evidence  of  the  following: 

Y  The  program  compares  favorably  to  other  programs 

with  similar  purpose  and  goals.  Programs  are  not  limited  to  Federal  government  and 
can  include  State  and  local  government  and  the  private  sector.  The  user  should 
consider  relevant  evaluations  and/or  data  that  are  collected  in  a  systematic  fashion 
that  allow  a  comparison  of  programs  with  similar  purpose  and  goals.  Comparisons 
must  include  an  assessment  of  the  most  significant  aspects  of  the  program’s 
performance. 

Not  Applicable:  A  Not  Applicable  rating  is  appropriate  if  1)  no  comparable  Federal,  state, 
local  government,  or  private  sector  programs  exist,  or  2)  the  comparison  would  be  too 
inherently  difficult  and  costly  to  perform  for  the  foreseeable  future.  The  explanation  for  Not 
Applicable  must  explain  why  the  comparison  is  inherently  too  difficult  to  perform. 

Additional  Guidance:  For  Capital  Assets  and  Service  Acquisition  programs,  there  are  a  few 
additional  considerations. 

■=>  For  Capital  Assets  and  Service  Acquisition  programs,  review  of  performance 
should  include  cost/schedule  adherence,  quality,  and  quantity  of  deliverables. 

Evidence/Data:  Evidence  can  include  comparative  performance  data  for  common 
performance  measures,  as  well  as  evaluations  and  documentation  comparing  similar 
programs. 

4.5:  Do  independent  evaluations  of  sufficient  scope  and  quality  indicate  that  the 
program  is  effective  and  achieving  results? 

Purpose:  To  detennine  whether  the  program  is  effective 
based  on  independent  and  comprehensive  evaluations.  This 
question  may  be  particularly  important  for  programs  that 
have  substantial  difficulty  formulating  quantitative 
performance  measures. 

For  R&D  programs,  this  question  is  central  to  retrospective  assessment  of  all  of  the  R&D 
criteria  (see  Appendix  C). 


Question  4.5:  Quick  Tips 


Answer  Yes,  Large  Extent, 

Options  Small  Extent,  No 


Question  4.4:  Quick  Tips 

Answer 

Options 

Yes,  Large  Extent, 
Small  Extent,  No, 

NA 

Additional 

Guidance 

•  Capital  Assets 
and  Service 
Acquisition 
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Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  the 
following: 

Y  Independent  program  evaluations  of  sufficient  scope  and  quality  indicate  that  the 
program  is  effective.  An  evaluation,  or  set  of  evaluations,  must  meet  the  quality, 
scope,  and  independence  criterion  as  defined  in  Question  2.6  to  be  considered 
evidence  for  this  question.  If  this  response  relies  on  different  evaluations  than  those 
used  for  2.6,  the  explanation  must  address  the  scope,  quality,  and  independence 
criteria. 

To  provide  evidence  of  results  an  evaluation  must  have  been  completed  or  at  a  minimum 
have  produced  some  interim  findings.  Unlike  in  Question  2.6,  a  program  cannot  receive 
credit  in  this  question  for  planned  evaluations. 

Relevant  evaluations  would  be  at  the  national  program  level,  rather  than  evaluations  of  one 
or  more  program  partners,  and  would  not  focus  only  on  process  indicators  such  as  the 
number  of  grants  provided,  or  hits  on  a  web  site. 

Not  Applicable:  Not  Applicable  is  not  an  option  for  this  question.  Given  the  flexibility  in 
determining  what  constitutes  an  evaluation,  all  programs  should  undergo  an  evaluation  that 
meets  the  elements  of  quality,  scope,  and  independence  detailed  in  Question  2.6. 

Evidence/Data:  Evidence  should  include  a  summary  discussion  of  the  findings  of  an 
evaluation  conducted  by  academic  and  research  institutions,  agency  contracts,  other 
independent  entities,  the  Government  Accountability  Office,  or  Inspectors  General. 


Specific  Results  Questions  by  Program  Type 

4.RG1:  Were  programmatic  goals  (and  benefits)  achieved  at  the  least 
incremental  societal  cost  and  did  the  program  maximize  net  benefits? 
(Regulatory) 

Purpose:  To  determine  whether  the  program  met  its  goals 
in  the  most  efficient  way  possible.  It  should  be  determined 
whether  the  program  achieved  the  anticipated  benefits 
through  implementation  of  its  significant  regulatory 
actions  (as  opposed  to  regulatory  development).  In 
calculating  the  incremental  costs  and  benefits  of 
regulations,  these  costs  should  be  compared  to  a  baseline  or,  in  a  small  number  of  cases,  a 
less  stringent  alternative.  This  question  deals  with  the  actual  implementation  of  the 
regulatory  action,  not  just  the  conception  and  promulgation  of  the  regulatory  action. 

Elements  of  Yes :  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  each  of  the 
following: 

Y  The  program  conducts  an  analysis  on  a  periodic  basis  (e.g.,  every  five  years)  of  the 
actual  effects  of  all  significant  regulations  or  a  subset  thereof  (e.g.,  economically 


Question  4.RG1:  Quick  Tips 


Answer 

Options 


Yes,  Large  Extent, 
Small  Extent,  No, 
NA 
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significant  regulatory  actions)  and  that  the  findings  meet  or  exceed  the  program’s 
original  estimates  of  the  net  benefits.  Programs  that  periodically  evaluate  the  effects 
of  their  regulations  or  rely  on  external  evaluations  and  find  that  actual  effects  are 
consistently  less  beneficial  than  anticipated  should  get  a  Small  Extent  to  Large  Extent. 

Evidence/Data:  Evidence  can  include: 

•  A  program  plan  or  process  to  conduct  this  exercise  on  a  regular  basis  such  as  the 
periodic  review  of  rules  that  impact  small  businesses  as  required  by  the  Regulatory 
Flexibility  Act,  Section  610; 

•  Documentation  (such  as  assessments,  independent  evaluations,  or  additional  impact 
analyses  using  retrospective  data)  generated  as  a  result  of  the  above  processes  or 
plans;  and 

•  Any  changes  made  to  or  eliminations  from  the  program  or  its  regulations,  or  a 
justification  that  no  change  is  needed,  resulting  from  this  type  of  systematic  review. 

4.CA1:  Were  program  goals  achieved  within  budgeted  costs  and  established 
schedules?  (Capital  Assets  and  Service  Acquisition) 

Purpose:  To  detennine  whether  valid  program  goals  were 
achieved  within  budgeted  costs  and  established  schedules 
and  whether  the  program  spends  funds  as  planned  and 
budgeted.  For  capital  assets  procured  and  in  operation,  this 
question  also  addresses  management-in-use  (i.e.,  the 
operations  and  disposal  costs). 

Elements  of  Yes:  A  Yes  answer  needs  to  clearly  explain  and  provide  evidence  of  the 
following: 

Y  The  program  achieved  the  goals  evaluated  in  Section  II  on  budget  and  on  schedule. 

An  example  of  a  program  that  could  receive  a  No  rating  could  be  an  acquisition  program  that 
has  experienced  60  percent  cost  growth  and  is  behind  schedule.  If  a  program’s  cost  and 
schedule  targets  were  changed  in  the  last  12  months  specifically  due  to  failure  to  achieve 
previous  goals,  the  program  should  get  a  No. 

Evidence/Data:  Evidence  can  include  a  comparison  of  the  contract  schedule,  deliverables, 
and  costs  with  the  final  outcomes  for  that  fiscal  year. 


Question  4.CA1:  Quick  Tips 


Answer 

Options 


Yes,  Large  Extent, 
Small  Extent,  No, 
NA 
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FINALIZING  THE  PART 


Entering  Information  in  PARTWeb 

PARTWeb  is  a  centralized,  web-based  tool  that  allows  for  governmentwide  collection  and 
reporting  of  PART  data,  including  management  recommendations  based  on  the  PART 
assessments.  The  PARTWeb  User  Manual  can  be  found  online 

(http://www.whitehouse.gov/omb/part/partweb/partweb  user  manual  07-2006.pdf)  and  specific 
directions  for  PARTWeb  are  issued  throughout  the  year  as  needed. 

Information  on  registering  for  PARTWeb  access  can  be  found  at  http://www.omb.gov/part. 


Determining  the  Rating 

The  answers  to  specific  questions  in  the  PART  translate  into  section  scores  which  are  weighted 
to  generate  an  overall  score.  The  section  scores  are  weighted  as  follows  to  generate  an  overall 
score:  program  purpose/design,  20  percent;  planning,  10  percent;  management,  20  percent;  and 
results/accountability,  50  percent.  PART  scores  are  translated  into  qualitative  ratings  based  on 
the  ranges  below.  Because  overall  scores  could  suggest  a  false  degree  of  precision,  only  the 
overall  ratings  are  made  available  to  the  public. 


Rating 

Range 

Effective 

85  -  100 

Moderately  Effective 

70-84 

Adequate 

50-69 

Ineffective 

0-  49 

Results  Not  Demonstrated 

Regardless  of  overall  score,  a  rating  of  Results  Not  Demonstrated  is  given  when  programs  do  not 
have  acceptable  long-term  and  annual  performance  measures.  A  program  also  gets  a  rating  of 
Results  Not  Demonstrated  when  it  lacks  baselines  and  performance  data  to  indicate  how  it  has 
been  performing. 

Generally,  a  program  that  has  a  No  on  question  2. 1  or  question  2.3  should  receive  a  Results  Not 
Demonstrated  rating.  A  program  that  received  Yes  to  these  questions,  but  has  a  No  on  questions 
4.1  and  4.2  because  it  does  not  have  data  on  its  performance  measures  (as  opposed  to  receiving  a 
No  due  to  failure  to  meet  its  goals)  should  also  receive  a  Results  Not  Demonstrated  rating. 
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Developing  an  Aggressive  Improvement  Plan 

In  response  to  each  PART  assessment,  the  agency  and  OMB  work  together  to  identify  follow-up 
actions  (also  known  as  an  improvement  plan)  to  improve  performance.  These  actions  can  include 
management  actions  the  agency  will  take,  funding  proposals  included  in  the  President’s  Budget, 
and  legislative  proposals.  Agency  efforts  to  complete  these  actions  and  improve  program 
performance  are  tracked  through  in  the  President’s  Management  Agenda  scorecard  for  Budget 
and  Perfonnance  Integration,  as  well  as  through  PART  updates  each  year. 

Rigorous  follow-up  actions  to  PART  findings  will  accelerate  improvements  in  Federal  programs. 
This  will  ensure  that  the  hard  work  done  through  the  PART  produces  perfonnance  and 
management  improvements. 

In  general,  OMB  and  the  agency  should  agree  upon  follow-up  actions  that  can  be  completed 
within  a  year.  If  the  planned  action  will  take  longer  than  a  year,  such  as  a  legislative 
recommendation  to  combine  the  program  with  others,  a  comment  to  that  effect  should  be  added 
to  that  action  on  the  Follow-Up  Actions  screen  in  PART  Web. 

Each  PART  follow-up  action  should  be  discrete,  and  OMB  and  the  agency  should  agree  in 
advance  to  how  they  will  know  the  action  has  been  completed.  As  follow-up  actions  are 
completed,  there  will  be  periodic  opportunities  to  update  the  status  in  PARTWeb  to  reflect  this 
accomplishment. 

In  addition,  as  programs  complete  their  follow-up  actions,  they  should  be  identifying  additional 
steps  they  will  take  to  continue  improving  the  perfonnance  of  their  programs.  The  updated  status 
of  the  improvement  plans  will  be  published  on  ExpectMore.gov  twice  a  year. 


Preparing  PART  Information  for  Publication 

ExpectM ore.gov 

ExpectMore.gov  was  launched  in  February  2006.  The  purpose  of  the  website  is  to  infonn  the 
public  about  program  perfonnance.  It  conveys  to  the  American  people  how  the  programs  they 
fund  are  performing  and  what  we  are  doing  to  improve  their  performance. 

ExpectMore.gov  identifies  which  programs  are  performing  and  which  are  not.  It  has  basic  search 
capability  that  facilitates  comparing  similar  programs.  Each  program  has  a  PART  summary  page 
that  describes  the  program’s  purpose  and  provides  key  findings  and  follow-up  actions.  From  the 
summary,  visitors  are  able  to  link  to  the  complete  PART  (answers,  explanations,  evidence,  and 
performance  information),  as  well  as  the  information  on  the  status  of  improvement  plans  and 
funding  information. 
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Naming  Programs 

Program  names  should  be  able  to  stand  alone  and  be  understood  independently.  Below  are  some 
guidelines  for  program  names  to  make  them  easier  for  the  public  to  understand. 

•  Avoid  using  acronyms  in  the  program  name.  If  the  program  name  must  use  an  acronym, 
it  should  be  introduced  in  the  “Program  Description”  section. 

•  Do  not  include  section  numbers  from  legislation  in  the  program  name. 

•  If  the  program  is  a  part  of  a  larger  agency,  do  not  state  the  agency  name  in  the  program 
name  unless  the  agency  name  is  necessary  to  differentiate  the  program  from  other 
agencies’  similar  programs  and  unless  the  agency  name  is  commonly  understood. 
Examples  of  agencies  that  are  commonly  understood  are  CDC  (Centers  for  Disease 
Control  and  Prevention),  FBI  (Federal  Bureau  of  Investigation),  and  FAA  (Federal 
Aviation  Administration). 

•  When  differentiating  between  programs  with  similar  names,  it  is  usually  more 
informative  to  include  some  words  to  describe  how  what  one  does  is  different  from  the 
other,  rather  than  just  stating  they  are  managed  by  different  agencies. 

General  Readability 

Since  the  individual  program  assessments  are  available  to  the  public  and  other  interested  parties 
through  ExpectMore.gov,  PART  summaries  and  the  complete  PARTs  must  be  written  in  clear, 
plain  English.  Acronyms  and  jargon  must  be  avoided  and  explanations  should  be  simple  and 
concise  enough  that  a  person  unfamiliar  with  the  program  would  understand  exactly  what  the 
program  is  getting  or  not  getting  credit  for.  Evidence  must  also  be  listed  in  a  clear  and 
understandable  way. 

Below  is  a  summary  of  key  guidelines: 

•  Keep  explanations  as  short  as  possible  while  maintaining  an  adequate  level  of 
information  to  justify  the  answer. 

•  Provide  evidence  of  the  judgment  used  in  determining  the  answer  instead  of  just  citing 
reports,  legislation,  or  regulations  in  the  evidence  sections.  Explanation  should  link  to  the 
evidence,  meaning  together  they  should  include  context  and  describe  the  relevant 
components  of  the  cited  documents  that  support  the  question  response. 

•  Where  possible,  present  specific  programmatic  details  and  outcomes  that  help  clarify 
responses  given,  rather  than  using  generalities. 

•  Be  careful  not  to  over-use  acronyms.  At  a  minimum,  state  the  full  title  at  least  the  first 
time  an  acronym  is  used.  Since  PARTs  are  multi -page  documents,  it  is  helpful  to  spell 
out  acronyms  more  frequently,  such  as  in  every  question  or  section. 

•  Avoid  the  use  personal  pronouns  (“we,”  “us,”  “they,”  etc). 

When  you  cite  a  web-based  resource  or  website,  include  the  full  URL  (http://...),  verify  that  the 
address  works,  and  be  as  specific  as  possible  about  what  you  are  referencing  (including  the  page 
number,  where  possible).  Web  addresses  are  hyperlinked  in  ExpectMore.gov,  allowing  the 
reader  to  click  on  the  link  and  review  the  evidence. 
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Drafting  Summaries 

PART  summaries  include  the  rating,  description,  findings,  and  set  of  follow-up  actions  (an 
“improvement  plan”).  When  you  enter  the  content  for  summaries  in  PARTWeb,  it  automatically 
formats  the  text  for  ExpectMore.gov. 

To  view  a  draft  of  the  summary  as  it  will  appear  on  ExpectMore.gov,  click  the  “PART 
Summary”  view  button  on  the  right  side  of  the  screen.  The  button  called  “Publishable  View”  will 
show  you  the  detailed  PART  information  that  will  be  made  available  to  the  public. 

Presenting  Measures  in  the  PART 

The  Measures  screen  in  PARTWeb  must  be  completed  for  all  PARTs,  and  measures  data  should 
support  questions  4. 1-4.3.  While  the  detailed  performance  information  need  not  be  included  in 
the  explanation  and  evidence  as  well,  any  performance  information  that  accompanies  the  answer 
should  be  consistent  with  the  measures  saved  in  PARTWeb.  Programs  cannot  receive  credit  for 
measures  that  are  not  included  in  PARTWeb;  measures  that  are  deemed  unacceptable  should  not 
be  listed  in  PARTWeb. 


For  more  information  on  using  PARTWeb,  see  the  user’s  manual 
(http://www.whitehouse.gov/omb/part/partweb/partweb  user  manual  07- 

2006.pdf). 


You  should  ensure  that  the  proper  categorization  is  selected  when  inputting  performance 
measures  into  the  Measures  page  in  PARTWeb.  When  entering  data  into  PARTWeb: 

•  Be  clear  what  the  units  are. 

•  Enter  whole  numbers  instead  of  increments  to  give  a  better  perspective  of  the  change. 

•  Do  not  put  “FY”  in  front  of  year. 

•  Present  the  measure  in  as  plain  English  as  possible  with  unit  specified.  For  example, 
“Percentage  of  program  participants  ...” 

•  For  measures  that  are  more  technical  in  nature,  provide  an  explanation  of  the  measures  in 
plain  English  and/or  provide  an  explanation  of  why  the  measure  is  useful  or  important  in 
the  program  context,  as  appropriate. 

In  addition,  you  must  set  targets  for  the  upcoming  budget  year  for  each  annual  measure  (i.e.,  at 
least  two  years  beyond  the  one  in  which  the  PART  is  completed). 

The  PART  summaries  posted  on  ExpectMore.gov  will  reflect  performance  infonnation,  inclusive 
of  these  measures,  and  will  link  to  the  full  PART  details  including  each  of  the  measures. 
Performance  data  and  targets  will  be  updated  at  least  once  a  year  to  ensure  information  is 
available  to  the  public  in  the  timeliest  fashion. 
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SUPPORTING  INFORMATION 


Reassessing  Programs 

When  a  program  provides  evidence  of  significant  improvement,  it  has  the  opportunity  to  be 
reassessed  so  that  it  may  also  improve  its  rating.  The  general  requirements  for  reassessments  are 
as  follows: 

•  Programs  may  be  reassessed  when  there  is  evidence  of  significant  change.  Examples  of 
significant  changes  include  those  that  would  warrant  changes  to  the  answers  of  multiple 
PART  questions  and  implementation  of  improved  performance  measures  (i.e.,  new 
measures  meet  the  standards  of  the  PART  and  there  are  adequate  data  to  set  targets  and 
demonstrate  results). 

•  When  requesting  reassessments,  Agencies  must  provide  both  of  the  following  for  the 
questions  they  believe  show  significant  improvement: 

o  draft  text  for  the  explanation  and  evidence 
o  copy  of  or  link  to  the  relevant  evidence  to  support  the  request 


PART  Crosscutting  Analysis 

Crosscutting  analyses  are  conducted  to  look  for  new  ways  to  improve  the  performance  of 
programs  with  similar  purpose  or  design.  Such  reviews  improve  coordination  and 
communication,  and  enable  program  managers  to  agree  to  common  goals  and  address  common 
challenges. 

Instead  of  combining  interdependent  programs  into  a  single  PART,  OMB  and  agencies  may 
compare  independent  programs  that  have  similar  purposes  or  goals  through  a  crosscutting 
analysis.  An  internal  crosscut  (programs  within  the  same  agency)  or  external  crosscut  (programs 
across  multiple  agencies)  would  identify  exemplary  goals  and  practices,  common  measures  of 
performance,  possible  tradeoffs  in  management  and  budget  decisions,  and/or  opportunities  for 
better  coordination  among  programs. 

Unlike  a  combined  PART  of  interdependent  programs,  a  crosscutting  analysis  would  examine 
programs  in  individual  PARTs.  Individual  assessments  allow  recognition  of  the  distinctive 
features  among  programs  while  a  crosscut  enables  similar  programs  to  explore  common  themes: 

•  What  is  the  target  population  for  each  program? 

•  What  products  and  services  are  provided  (common  output  measures  as  appropriate)? 

•  How  well  are  products  and  services  provided  (common  output  efficiency  measures  as 
appropriate)? 
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•  What  impact  have  the  programs  achieved  (common  outcome  measures  and  common 
outcome  efficiency  measures  as  appropriate)? 

An  internal  crosscut  need  not  require  production  of  additional  materials,  as  each  of  the 
aforementioned  corresponds  to  questions  already  posed  in  the  PART.  Individual  PART 
summaries  and  improvement  plans  would  reference  results  of  crosscutting  issues  as  appropriate. 

The  results  of  external  crosscutting  analyses,  however,  would  summarize  common  strengths  and 
opportunities  for  improvement,  drawing  upon  the  individual  PART  assessments.  Ultimately,  the 
analysis  could  lead  to  a  common  framework  of  performance  measures  and  accountability  if  one 
does  not  already  exist.  See  the  Analytical  Perspectives  for  examples  of  crosscutting  analyses  that 
have  been  completed  in  previous  years  (http ://www . whitehouse .gov/ omb/budget/ fy2 00 8) . 

OMB  and  agencies  are  encouraged  to  identify  opportunities  to  conduct  internal  and  external 
crosscutting  analyses. 


Budget  and  Performance  Integration  and 
the  Government  Performance  and  Results  Act 

PART  and  the  B PI  Initiative 

The  role  of  the  Budget  and  Performance  Integration  (BPI)  Initiative  is  to  ensure  that  Federal 
dollars  produce  the  greatest  results.  To  accomplish  this,  agencies  and  OMB  identify  which 
programs  work,  which  are  deficient,  and  what  can  be  done  to  improve  the  performance  of  them 
all.  This  is  done  through  the  PART. 

The  BPI  Initiative  aims  to  achieve  two  key  goals: 

1.  Improved  Program  Performance:  Through  the  use  of  PART  performance  assessments, 
programs  will  have  the  information  they  need  to  improve  their  performance  every  year. 
The  initiative  requires  each  agency  to  identify  opportunities  to  improve  program 
management  and  design,  and  then  develop  and  implement  clear,  aggressive  plans  to  get 
more  for  tax  dollars  every  year. 

2.  Greater  Investment  in  Successful  Programs:  Overall,  scarce  resources  need  to  be 
allocated  to  programs  that  benefit  the  Nation  most  effectively  and  efficiently.  Program 
performance  will  not  be  the  only  factor  in  decisions  about  how  much  funding  programs 
receive.  However,  the  Congress  and  the  President,  equipped  with  information  about 
program  performance  can  consider  performance  to  a  greater  degree  in  their  decision¬ 
making  and  invest  primarily  in  programs  that  provide  the  greatest  return  on  investment 
of  taxpayer  dollars.  If  poor  performing  programs  are  unable  to  demonstrate  improved 
results,  then  that  investment  may  be  reallocated  to  programs  that  can  demonstrate  greater 
success. 

The  BPI  Initiative  scorecard  standards  are  in  Appendix  A. 
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PART  and  the  Government  Performance  and  Results  Act 

The  PART  is  a  vehicle  for  achieving  the  goals  of  the  Government  Performance  and  Results  Act 
(GPRA).  Section  2(b)  of  the  Government  Performance  and  Results  Act  of  1993  identified  the 
following  purposes  of  the  Act: 

1 .  Improve  the  confidence  of  the  American  people  in  the  capability  of  the  Federal 
Government,  by  systematically  holding  Federal  agencies  accountable  for  achieving 
program  results; 

2.  Initiate  program  performance  refonn  with  a  series  of  pilot  projects  in  setting  program 
goals,  measuring  program  performance  against  those  goals,  and  reporting  publicly  on 
their  progress; 

3.  Improve  Federal  program  effectiveness  and  public  accountability  by  promoting  a  new 
focus  on  results,  service  quality,  and  customer  satisfaction; 

4.  Help  Federal  managers  improve  service  delivery,  by  requiring  that  they  plan  for  meeting 
program  objectives  and  by  providing  them  with  information  about  program  results  and 
service  quality; 

5.  Improve  Congressional  decision-making  by  providing  more  objective  information  on 
achieving  statutory  objectives,  and  on  the  relative  effectiveness  and  efficiency  of  Federal 
programs  and  spending;  and 

6.  Improve  internal  management  of  the  Federal  Government. 

The  PART  aims  to  achieve  all  these  purposes.  For  example,  the  FY  2007  Budget  launched 
ExpectMore.gov,  a  website  aimed  specifically  at  making  performance  information  transparent 
and  readily  available  to  the  American  people  (i.e.,  the  first  purpose). 

The  PART  strengthens  and  reinforces  performance  measurement  under  GPRA  by  encouraging 
careful  development  of  perfonnance  measures  according  to  the  outcome-oriented  standards  of 
the  law  and  by  requiring  that  agency  goals  be  appropriately  ambitious.  Performance  measures 
included  in  GPRA  plans  and  reports  and  those  developed  or  revised  through  the  PART 
process  must  be  consistent. 

The  PART  process  should  also  help  agencies  develop  and  identify  meaningful  perfonnance 
measures  to  support  their  GPRA  reporting.  When  developing  perfonnance  measures,  agencies 
should  consult  with  stakeholders,  as  appropriate. 

Agency  performance  plans  and  reports  vary  in  their  level  of  detail  and  how  they  address  the 
different  programs  the  agency  operates.  When  annual  plans  and  reports  include  programs  that 
have  been  assessed  in  the  PART,  the  measures  used  for  GPRA  should  be  the  same  as  those 
included  in  the  PART.  In  all  cases,  performance  measures  included  in  GPRA  plans  and  reports 
should  meet  the  standards  of  the  PART  -  they  must  be  outcome  oriented,  relate  to  the  overall 
purpose  of  the  program,  and  have  ambitious  targets. 
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Agencies  that  use  performance  information  to  manage  necessarily  rely  on  numerous  types  of 
data  and  measures  to  infonn  and  monitor  their  actions.  While  output  measures  are  useful  for  day 
to  day  measurement,  they  are  generally  not  acceptable  for  the  PART. 

While  the  PART  process  provides  a  useful  vehicle  for  developing  new  performance  measures, 
agencies  need  not  wait  until  a  program  is  being  assessed  to  develop  or  improve  its  performance 
measures  and  begin  collecting  data. 
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APPENDIX  A:  BUDGET  AND  PERFORMANCE  INTEGRATION 

INITIATIVE  CRITERIA 


Progress  on  the  President’s  Management  Agenda  (PMA)  is  assessed  quarterly  using  a  scorecard. 
The  Executive  Branch  Management  Scorecard  tracks  how  well  the  departments  and  major 
agencies  are  executing  the  five  government-wide  management  initiatives,  including  Budget  and 
Performance  Integration  (BPI). 

The  PART  is  central  to  the  Administration’s  BPI  Initiative  because  it  drives  a  sustained  focus  on 
results.  To  earn  a  high  PART  rating,  a  program  must  use  performance  to  manage,  justify  its 
resource  requests  based  on  the  performance  it  expects  to  achieve,  and  continually  improve 
efficiency  -  all  goals  of  the  BPI  Initiative. 

The  current  BPI  scorecard  standards  for  success,  as  well  as  the  standards  for  all  of  the  other 
PMA  Initiatives,  can  be  seen  online:  http ://www . whitehouse . gov/results/ agenda/ standards .p df. 
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APPENDIX  B:  ADDITIONAL  RESOURCES  ON 
RIGOROUS  PROGRAM  EVALUATIONS 


The  following  are  several  links  to  resources  on  program  evaluation  located  on  the  internet  that 
may  be  helpful.  This  is  not  intended  to  be  an  exhaustive  list. 

■  Coalition  for  Evidence-Based  Policy;  Evidence-Based  Policy  Help  Desk,  (a  web  site) 
http://www.evidencebasedpolicy.org/ 

■  Program  Evaluation  Methods:  Measurement  and  Attribution  of  Program  Results;  Treasury 
Board  of  Canada,  Secretariat;  1998.  (a  book  available  online) 
http://www.tbs-sct.gc.ca/eval/pubs/meth/pem-mep  e.pdf 

■  Understanding  Impact  Evaluation;  The  World  Bank  Group,  (a  web  site) 
http://www.worldbank.org/poverty/impact/index.htm 

■  “Program  Evaluation:  An  Evaluation  Culture  and  Collaborative  Partnerships  Build 
Agency  Capacity;”  GAO-03-454;  Government  Accountability  Office;  May  2003. 
http://www.gao.gov/docdblite/summary.php?recflag=&accno=A06797&rptno=GAO-03- 

454 


■  “Performance  Measurement  and  Evaluation:  Definitions  and  Relationships;”  GAO/GGD- 
98-26;  Government  Accountability  Office;  April  1998. 

http://www.gao.gov/docdblite/summarv.php?recflag=&accno=160204&rptno=GGD-98-26 

■  “Designing  Evaluations;”  GAO/PEMD-1 0.1.4;  Government  Accountability  Office;  May 
1991.  http://161.203.16.4/t2pbat7/144040.pdf 

■  Research  Methods  Knowledge  Base;  Trochim,  William  M.;  Cornell  University. 

(a  web  site)  http://www.socialresearchmethods.net/kb/ 

■  “Identifying  and  Implementing  Educational  Practices  Supported  By  Rigorous  Evidence:  A 
User  Friendly  Guide;”  U.S.  Department  of  Education;  December  2003. 
http://www.ed.gov/rschstat/research/pubs/rigorousevid/rigorousevid.pdf 
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APPENDIX  C:  RESEARCH  AND  DEVELOPMENT  PROGRAM 

INVESTMENT  CRITERIA 

As  another  initiative  of  the  President’s  Management  Agenda,  the  development  of  explicit  R&D 
investment  criteria  builds  on  the  best  of  the  planning  and  assessment  practices  that  R&D 
program  managers  use  to  plan  and  assess  their  programs.  The  Administration  has  worked  with 
experts  and  stakeholders  to  build  upon  lessons  learned  from  previous  approaches. 

Agencies  should  use  the  criteria  as  broad  guidelines  that  apply  at  all  levels  of  Federally  funded 
R&D  efforts,  and  they  should  use  the  PART  as  the  instrument  to  periodically  evaluate 
compliance  with  the  criteria  at  the  program  level.  To  make  this  possible,  the  R&D  PART  aligns 
with  the  R&D  criteria.  The  R&D  criteria  are  reprinted  here  as  a  guiding  framework  for 
addressing  the  R&D  PART. 

The  R&D  criteria  address  not  only  planning,  management,  and  prospective  assessment  but  also 
retrospective  assessment.  Retrospective  review  of  whether  investments  were  well-directed, 
efficient,  and  productive  is  essential  for  validating  program  design  and  instilling  confidence  that 
future  investments  will  be  wisely  invested.  Retrospective  reviews  should  address  continuing 
program  relevance,  quality,  and  successful  performance  to  date. 

While  the  criteria  are  intended  to  apply  to  all  types  of  R&D,  the  Administration  is  aware  that 
predicting  and  assessing  the  outcomes  of  basic  research  in  particular  is  never  easy.  Serendipitous 
results  are  often  the  most  interesting  and  ultimately  may  have  the  most  value.  Taking  risks  and 
working  toward  difficult-to-attain  goals  are  important  aspects  of  good  research  management,  and 
innovation  and  breakthroughs  are  among  the  results.  However,  there  is  no  inherent  conflict 
between  these  facts  and  a  call  for  clearer  infonnation  about  program  goals  and  perfonnance 
toward  achieving  those  goals.  The  Administration  expects  agencies  to  focus  on  improving  the 
management  of  their  research  programs  and  adopting  effective  practices,  and  not  on  predicting 
the  unpredictable. 

The  R&D  investment  criteria  have  several  potential  benefits: 

•  Use  of  the  criteria  allows  policy  makers  to  make  decisions  about  programs  based  on 
information  beyond  anecdotes,  prior-year  funding  levels,  and  lobbying  of  special 
interests. 

•  A  dedicated  effort  to  improve  the  process  for  budgeting,  selecting,  and  managing  R&D 
programs  is  helping  to  increase  the  return  on  taxpayer  investment  and  the  productivity  of 
the  Federal  R&D  portfolio. 

•  The  R&D  investment  criteria  will  help  communicate  the  Administration’s  expectations 
for  proper  program  management. 

•  The  criteria  and  subsequent  implementation  guidance  will  also  set  standards  for 
information  to  be  provided  in  program  plans  and  budget  justifications. 
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The  processes  and  collected  infonnation  promoted  under  the  criteria  will  improve  public 
understanding  of  the  possible  benefits  and  effectiveness  of  the  Federal  investment  in 


R&D. 


Details  on  the  Criteria 

The  Relevance,  Quality,  and  Perfonnance  criteria  apply  to  all  R&D  programs.  Industry-  or 
market-relevant  applied  R&D  must  meet  additional  criteria.  Together,  these  criteria  can  be  used 
to  assess  the  need,  relevance,  appropriateness,  quality,  and  performance  of  Federal  R&D 
programs. 


I.  Relevance 

R&D  investments  must  have  clear  plans,  must  be  relevant  to  national  priorities,  agency  missions, 
relevant  fields,  and  “customer”  needs,  and  must  justify  their  claim  on  taxpayer  resources. 
Programs  that  directly  support  Presidential  priorities  may  receive  special  consideration  with 
adequate  documentation  of  their  relevance.  Review  committees  should  assess  program  objectives 
and  goals  on  their  relevance  to  national  needs,  “customer”  needs,  agency  missions,  and  the 
field(s)  of  study  the  program  strives  to  address.  For  example,  the  Joint  DOE/NSF  Nuclear 
Sciences  Advisory  Committee’s  Long  Range  Plan  and  the  Astronomy  Decadal  Surveys  are  the 
products  of  good  planning  processes  because  they  articulate  goals  and  priorities  for  research 
opportunities  within  and  across  their  respective  fields. 

OMB  will  work  with  some  programs  to  identify  quantitative  metrics  to  estimate  and  compare 
potential  benefits  across  programs  with  similar  goals.  Such  comparisons  may  be  within  an 
agency  or  among  agencies. 


A.  Programs  must  have  complete  plans,  with  clear  goals  and  priorities. 

Programs  must  provide  complete  plans,  which  include  explicit  statements  of: 

-  specific  issues  motivating  the  program; 

-  broad  goals  and  more  specific  tasks  meant  to  address  the  issues; 

-  priorities  among  goals  and  activities  within  the  program; 

-  human  and  capital  resources  anticipated;  and 

-  intended  program  outcomes,  against  which  success  may  later  be  assessed. 

B.  Programs  must  articulate  the  potential  public  benefits  of  the  program. 

Programs  must  identify  potential  benefits,  including  added  benefits  beyond  those  of  any 
similar  efforts  that  have  been  or  are  being  funded  by  the  government  or  others.  R&D 
benefits  may  include  technologies  and  methods  that  could  provide  new  options  in  the 
future,  if  the  landscape  of  today’s  needs  and  capabilities  changes  dramatically.  Some 
programs  and  sub-program  units  may  be  required  to  quantitatively  estimate  expected 
benefits,  which  would  include  metrics  to  permit  meaningful  comparisons  among 
programs  that  promise  similar  benefits.  While  all  programs  should  try  to  articulate 
potential  benefits,  OMB  and  OSTP  recognize  the  difficulty  in  predicting  the  outcomes  of 
basic  research.  Consequently,  agencies  may  be  allowed  to  relax  this  as  a  requirement  for 
basic  research  programs. 
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C.  Programs  must  document  their  relevance  to  specific  Presidential  priorities  to 
receive  special  consideration. 

Many  areas  of  research  warrant  some  level  of  Federal  funding.  Nonetheless,  the  President 
has  identified  a  few  specific  areas  of  research  that  are  particularly  important.  To  the 
extent  a  proposed  project  can  document  how  it  directly  addresses  one  of  these  areas,  it 
may  be  given  preferential  treatment. 

D.  Program  relevance  to  the  needs  of  the  Nation,  of  fields  of  Science  &  Technology,  and 
of  program  “customers”  must  be  assessed  through  prospective  external  review. 

Programs  must  be  assessed  on  their  relevance  to  agency  missions,  fields  of  science  or 
technology,  or  other  “customer”  needs.  A  customer  may  be  another  program  at  the  same 
or  another  agency,  an  interagency  initiative  or  partnership,  or  a  firm  or  other  organization 
from  another  sector  or  country.  As  appropriate,  programs  must  define  a  plan  for  regular 
reviews  by  primary  customers  of  the  program’s  relevance  to  their  needs.  These  programs 
must  provide  a  plan  for  addressing  the  conclusions  of  external  reviews. 

E.  Program  relevance  to  the  needs  of  the  Nation,  of  fields  of  S&T,  and  of  program 
“customers”  must  be  assessed  periodically  through  retrospective  external  review. 

Programs  must  periodically  assess  the  need  for  the  program  and  its  relevance  to 
customers  against  the  original  justifications.  Programs  must  provide  a  plan  for  addressing 
the  conclusions  of  external  reviews. 


II.  Quality 

Programs  should  maximize  the  quality  of  the  R&D  they  fund  through  the  use  of  a  clearly  stated, 
defensible  method  for  awarding  a  significant  majority  of  their  funding.  A  customary  method  for 
promoting  R&D  quality  is  the  use  of  a  competitive,  merit-based  process.  NSF’s  process  for  the 
peer-reviewed,  competitive  award  of  its  R&D  grants  is  a  good  example.  Justifications  for 
processes  other  than  competitive  merit  review  may  include  “outside -the-box”  thinking,  a  need 
for  timeliness  (e.g.,  R&D  grants  for  rapid  response  studies  of  Pfisteria),  unique  skills  or 
facilities,  or  a  proven  record  of  outstanding  perfonnance  (e.g.,  performance-based  renewals). 

Programs  must  assess  and  report  on  the  quality  of  current  and  past  R&D.  For  example,  NSF’s 
use  of  Committees  of  Visitors,  which  review  NSF  directorates,  is  an  example  of  a  good  quality- 
assessment  tool.  OMB  and  OSTP  encourage  agencies  to  provide  the  means  by  which  their 
programs  may  be  benchmarked  internationally  or  across  agencies,  which  provides  one  indicator 
of  program  quality. 


A.  Programs  allocating  funds  through  means  other  than  a  competitive,  merit-based 
process  must  justify  funding  methods  and  document  how  quality  is  maintained. 

Programs  must  clearly  describe  how  much  of  the  requested  funding  will  be  broadly 
competitive  based  on  merit,  providing  compelling  justifications  for  R&D  funding 
allocated  through  other  means.  (See  OMB  Circular  A-l  1  for  definitions  of  competitive 
merit  review  and  other  means  of  allocating  Federal  research  funding.)  All  program  funds 
allocated  through  means  other  than  unlimited  competition  must  document  the  processes 
they  will  use  to  distribute  funds  to  each  type  of  R&D  perfonner  (e.g.,  Federal 
laboratories,  Federally-funded  R&D  centers,  universities,  etc.).  Programs  are  encouraged 
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to  use  external  assessment  of  the  methods  they  use  to  allocate  R&D  and  maintain 
program  quality. 

B.  Program  quality  must  be  assessed  periodically  through  retrospective  expert  review. 

Programs  must  institute  a  plan  for  regular,  external  reviews  of  the  quality  of  the 
program’s  research  and  research  performers,  including  a  plan  to  use  the  results  from  these 
reviews  to  guide  future  program  decisions.  Rolling  reviews  performed  every  3-5  years  by 
advisory  committees  can  satisfy  this  requirement.  Benchmarking  of  scientific  leadership 
and  other  factors  provides  an  effective  means  of  assessing  program  quality  relative  to 
other  programs,  other  agencies,  and  other  countries. 


III.  Performance 

R&D  programs  should  maintain  a  set  of  high  priority,  multi-year  R&D  objectives  with  annual 
performance  outputs  and  milestones  that  show  how  one  or  more  outcomes  will  be  reached. 
Metrics  should  be  defined  not  only  to  encourage  individual  program  performance  but  also  to 
promote,  as  appropriate,  broader  goals,  such  as  innovation,  cooperation,  education,  and 
dissemination  of  knowledge,  applications,  or  tools. 

OMB  encourages  agencies  to  make  the  processes  they  use  to  satisfy  the  Government 
Perfonnance  and  Results  Act  (GRP A)  consistent  with  the  goals  and  metrics  they  use  to  satisfy 
these  R&D  criteria.  Satisfying  the  R&D  performance  criteria  for  a  given  program  should  serve  to 
set  and  evaluate  R&D  performance  goals  for  the  purposes  of  GPRA.  OMB  expects  goals  and 
performance  measures  that  satisfy  the  R&D  criteria  to  be  reflected  in  agency  perfonnance  plans. 

Programs  must  demonstrate  an  ability  to  manage  in  a  manner  that  produces  identifiable  results. 
At  the  same  time,  taking  risks  and  working  toward  difficult-to-attain  goals  are  important  aspects 
of  good  research  management,  especially  for  basic  research.  The  intent  of  the  investment  criteria 
is  not  to  drive  basic  research  programs  to  pursue  less  risky  research  that  has  a  greater  chance  of 
success.  Instead,  the  Administration  will  focus  on  improving  the  management  of  basic  research 
programs. 

OMB  will  work  with  some  programs  to  identify  quantitative  metrics  to  compare  performance 
across  programs  with  similar  goals.  Such  comparisons  may  be  within  an  agency  or  among 
agencies. 

Construction  projects  and  facility  operations  will  require  additional  performance  metrics.  Cost 
and  schedule  earned-value  metrics  for  the  construction  of  R&D  facilities  must  be  tracked  and 
reported.  Within  DOE,  the  Office  of  Science’s  fonnalized  independent  reviews  of  technical  cost, 
scope,  and  schedule  baselines  and  project  management  of  construction  projects  (“Lehman 
Reviews”)  are  widely  recognized  as  an  effective  practice  for  discovering  and  correcting 
problems  involved  with  complex,  one-of-a-kind  construction  projects. 
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A.  Programs  may  be  required  to  track  and  report  relevant  program  inputs  annually. 

Programs  may  be  expected  to  report  relevant  program  inputs,  which  could  include 
statistics  on  overhead,  intramural/extramural  spending,  infrastructure,  and  human  capital. 
These  inputs  should  be  discussed  with  OMB. 

B.  Programs  must  define  appropriate  output  and  outcome  measures,  schedules,  and 
decision  points. 

Programs  must  provide  single-  and  multi-year  R&D  objectives,  with  annual  performance 
outputs,  to  track  how  the  program  will  improve  scientific  understanding  and  its 
application.  Programs  must  provide  schedules  with  annual  milestones  for  future 
competitions,  decisions,  and  tennination  points,  highlighting  changes  from  previous 
schedules.  Program  proposals  must  define  what  would  be  a  minimally  effective  program 
and  a  successful  program.  Agencies  should  define  appropriate  output  and  outcome 
measures  for  all  R&D  programs,  but  agencies  should  not  expect  fundamental  basic 
research  to  be  able  to  identify  outcomes  and  measure  performance  in  the  same  way  that 
applied  research  or  development  are  able  to.  Highlighting  the  results  of  basic  research  is 
important,  but  it  should  not  come  at  the  expense  of  risk-taking  and  innovation.  For  some 
basic  research  programs,  OMB  may  accept  the  use  of  qualitative  outcome  measures  and 
quantitative  process  metrics.  Facilities  programs  must  define  metrics  and  methods  (e.g., 
earned-value  reporting)  to  track  development  costs  and  to  assess  the  use  and  needs  of 
operational  facilities  over  time.  If  leadership  in  a  particular  field  is  a  goal  for  a  program 
or  agency,  OMB  and  OSTP  encourage  the  use  of  benchmarks  to  assess  the  processes  and 
outcomes  of  the  program  with  respect  to  leadership.  OMB  encourages  agencies  to  make 
the  processes  they  use  to  satisfy  GPRA  consistent  with  the  goals  and  metrics  they  use  to 
satisfy  these  R&D  criteria. 

C.  Program  performance  must  be  retrospectively  documented  annually. 

Programs  must  document  performance  against  previously  defined  output  and  outcome 
metrics,  including  progress  toward  objectives,  decisions,  and  termination  points  or  other 
transitions.  Programs  with  similar  goals  may  be  compared  on  the  basis  of  their 
performance.  OMB  will  work  with  agencies  to  identify  such  programs  and  appropriate 
metrics  to  enable  such  comparisons. 


IV.  Criteria  for  R&D  Programs  Developing  Technologies  That  Address  Industry 
Issues 

The  purpose  of  some  R&D  and  technology  demonstration  programs  and  projects  is  to  introduce 
some  product  or  concept  into  the  marketplace.  However,  some  of  these  efforts  engage  in 
activities  that  industry  is  capable  of  doing  and  may  discourage  or  even  displace  industry 
investment  that  would  occur  otherwise.  For  the  purposes  of  assessing  Federal  R&D  investments, 
the  following  criteria  should  be  used  to  assess  industry-relevant  R&D  and  demonstration 
projects,  including,  at  OMB  discretion,  associated  construction  activities. 

OMB  will  work  with  programs  to  identify  quantitative  metrics  to  measure  and  compare  potential 
benefits  and  perfonnance  across  programs  with  similar  goals,  as  well  as  ways  to  assess  market 
relevance. 
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A.  Programs  and  projects  must  articulate  public  benefits  of  the  program  using  uniform 
benefit  indicators  across  programs  and  projects  with  similar  goals. 

In  addition  to  the  public  benefits  required  in  the  general  criteria,  all  industry-relevant 
programs  and  projects  must  identify  and  use  unifonn  benefit  indicators  (including 
benefit-cost  ratios)  to  enable  comparisons  of  expected  benefits  across  programs  and 
projects.  OMB  will  work  with  agencies  to  identify  these  indicators. 

B.  Programs  and  projects  must  justify  the  appropriateness  of  Federal  investment, 
including  the  manner  in  which  the  market  fails  to  motivate  private  sector 
investment. 

A  lack  of  market  incentives  discourages  private  firms  from  investing  in  research  where 
the  benefits  may  occur  far  in  the  future,  the  risks  may  be  too  great  for  non-Federal 
participants,  or  the  benefits  accrue  to  the  public  rather  than  private  investors.  Programs 
and  projects  must  demonstrate  that  industry  investment  is  sub-optimal  and  explain  in 
what  way  the  market  fails  that  prevents  the  private  sector  from  capturing  the  benefits  of 
developing  the  good  or  service. 

C.  Programs  and  projects  must  demonstrate  that  investment  in  R&D  and 
demonstration  activities  is  the  best  means  to  support  the  Federal  policy  goals, 
compared  to  other  policy  alternatives. 

When  the  Federal  government  chooses  to  intervene  to  address  market  failures,  there  may 
be  many  policy  alternatives  to  address  those  failures.  Among  the  other  tools  available  to 
the  government  are  legislation,  tax  policy,  regulatory  and  enforcement  efforts,  and  an 
integrated  combination  of  these  approaches.  In  this  context,  projects  to  address  issues  of 
genuine  Federal  concern  should  be  able  to  illustrate  how  R&D  and  demonstration 
activities  are  superior  to  other  policy  tools  in  addressing  Federal  goals,  either  by 
themselves  or  as  part  of  an  integrated  package. 

D.  Programs  and  projects  must  document  industry  or  market  relevance,  including 
readiness  of  the  market  to  adopt  technologies  or  other  outputs. 

Programs  must  assess  the  likelihood  that  the  target  industry  will  be  able  to  adopt  the 
technology  or  other  program  outputs.  The  level  of  industry  cost  sharing  is  one  indicator 
of  industry  relevance.  Before  projects  move  into  demonstration  or  deployment  stages,  an 
economic  analysis  of  the  public  and  private  returns  on  the  public  investment  must  be 
provided. 

E.  Program  performance  plans  and  reports  must  include  “off  ramps”  and  transition 
points. 

In  addition  to  the  schedules  and  decision  points  defined  in  the  general  criteria,  program 
plans  should  also  identify  whether,  when,  and  how  aspects  of  the  program  may  be  shifted 
to  the  private  sector. 
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APPENDIX  D:  TWO  LIMITATIONS  OF  INPUT 
PRODUCTIVITY  MEASURES  USED  AS  OUTPUT  EFFICIENCY 

MEASURES 

This  appendix  discusses  two  limitations  of  using  input  productivity  measures  as  output  efficiency 
measures  for  the  PART.  The  measures  are  simply  defined  as  the  ratio  of  a  program’s  output  to 
an  input.  In  the  first  case,  because  the  PART  requires  comparisons  of  efficiency  measures 
between  different  time  periods,  it  is  important  to  see  how  outputs  produced  during  the 
measurement  periods  affect  the  validity  of  an  input  productivity  measure. 

Most  often,  changes  in  an  input  productivity  measure  are  used  to  indicate  improvement  or 
degradation  in  program  efficiency.  This  approach  is  applicable  to  a  program  that  produces  the 
same  or  similar  outputs  repetitively.  The  second  case  addresses  programs  that  produce  unique 
outputs. 

Case  1:  When  a  program ’s  outputs  or  output  mixes  are  different  in  the  time 
periods  compared. 

The  following  is  an  illustrative  situation  that  shows  how  a  simple  input  productivity  measure 
provides  an  inaccurate  indication  of  an  organization’s  efficiency:  Suppose  a  unit  of  a  law 
enforcement  agency  with  100  employees  performs  two  types  of  investigations  -  one  type  that 
trained  personnel  takes  40  labor-hours  to  complete  and  a  second  type  that  takes  them  80  labor- 
hours  to  complete  on  average. 

If  the  agency  uses  investigations  completed  per  employee  as  an  output  efficiency  measure  for 
employee  productivity  then  when  the  unit  is  being  100%  efficient  in  utilizing  its  capacity,  during 
an  80  work-hour  pay  period,  it  produces  2  investigations  per  employee  when  the  unit  solely 
processes  investigations  for  the  first  kind,  but  only  1  investigation  per  employee  when  it  solely 
processes  investigations  of  the  other  kind.  The  number  and  mix  of  investigations  the  unit  works 
on  varies  substantially  from  week-to-week  because  the  case  load  coming  to  the  unit  is 
unpredictable.  Below  is  an  example  tabulation  (Table  1)  of  investigations  completed  for  two  pay 
periods  and  the  computation  of  output  per  employee.  (Note:  While  the  table  below  uses  pay 
period  data  for  illustrative  purposes,  data  in  the  PART  would  typically  be  compiled  to  reflect 
annual  performance.) 


Table  1 :  Unit’s  Perfonnance  -  Investigations  completed  per  employee. 


Pay  Period  3 

Pay  Period  4 

(la)  Total  employees 

100 

100 

Type  1  Investigations:  (2a)  Number  completed 

95 

76 

Type  2  Investigations:  (3a)  Number  completed 

51 

61 

Total  Investigations:  (4a)  Number  completed 
(sum  of  boxes  2a  and  3a) 

146 

137 

(5  a)  Investigations  per  employee 

1.46 

1.37 
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Based  on  the  investigations  completed  per  employee  measure,  the  unit’s  labor  productivity 
decreased  by  0.09  investigations  per  employee  in  pay  period  4  compared  with  pay  period  3  -  a 
6%  decrease.  However,  note  that  the  work  performed  during  the  pay  periods  was  different. 
Though  the  unit  completed  fewer  investigations  during  pay  period  4,  it  completed  more  of  the 
second  investigation  type,  which,  on  average,  takes  twice  as  long  to  complete.  Since  the  work 
performed  is  not  comparable,  the  investigations  per  employee  measure  between  the  two  pay 
periods  is  also  not  comparable.  What  can  be  done? 

One  approach  is  to  use  a  standard  or  a  benchmark  to  define  what  would  happen  if  the  unit  was 
100%  efficient  given  its  actual  output  and  compare  it  with  actual  performance  over  a  pay  period. 
In  this  hypothetical  example,  we  assumed  that  a  study  provided  data  determining  the  amount  of 
effort  (labor-hours)  trained  personnel  require  to  complete  each  investigation  type.  The  standard 
generates  what  100%  efficiency  means  by  crediting  the  unit  for  labor-hours  earned  -  40  labor- 
hours  for  each  completed  investigation  of  the  first  type  and  80  labors-hours  for  the  second  type. 
The  actual  labor-hours  expended  by  the  unit  are  then  compared  against  the  actual  labor-hours 
the  unit  paid  for.  To  keep  the  illustration  simple,  suppose  the  unit’s  100  employees  work  80 
hours  during  a  full  pay  period  and,  thus,  have  8000  labor-hours  available.  Consequently,  the 
efficient  unit  has  the  capacity  during  a  pay  period  to  complete  200  investigations  of  the  first  type 
or  only  100  of  the  second  type.  But  to  complicate  matters  a  little,  the  unit  paid  for  20  and  60 
hours  of  overtime  in  the  third  and  fourth  pay  period,  respectively.  Using  this  approach,  the 
situation,  from  an  efficiency  standpoint  looks  much  different  because  the  data  show  efficiency 
for  both  pay  periods  were  essentially  the  same.  See  Table  2  below. 


Table  2:  Unit’s  perfonnance  based  on  a  labor-hour  standard. 


Pay  ~ 

3eriod  3 

Pay 

Period  4 

(la)  Total  employees  (lb)  Total  labor-hours 
paid  including  overtime 

100 

8020 

100 

8060 

Type  1  Investigations:  (2a)  Number  completed 
(2b)  Earned  labor-hours  (40  labor-hours 
times  box  2a) 

95 

3800 

76 

3040 

Type  2  Investigations:  (3a)  Number  completed 
(3b)  Earned  labor-hours  for  Type  2 
Investigations  (80  labor-hours  times  box  3a) 

51 

4080 

61 

4880 

All  Investigations:  (4a)  Number  completed  (sum 
of  boxes  2a  and  3a)  (4b)  Total  labor-hours 
earned  (sum  of  boxes  2b  and  3b) 

146 

7880 

137 

7920 

(5b)  Labor-hour  variance  (difference  between 
box  (lb)  and  (4b) 

+140 

+140 

(6a)  Investigations  per  employee  (6b)  Labor- 
hour  efficiency  (box  4a  divided  by  box  1  times 
100%) 

1.46 

98.25% 

1.37 

98.26% 
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Also  note  the  following  observations: 


o  The  above  example  “solution”  requires  more  data  than  the  input  measure  alternative. 
In  particular,  it  requires  a  study  to  establish  standards  and  accurate  timekeeping  of 
employees  by  the  unit.  However,  many  work  processes  do  not  lend  themselves  to 
standard-setting. 

o  The  efficiency  measure  is  sensitive  to  the  labor-hour  standards  used  to  generate  the 
hypothetical  100%  efficiency  case  that  actual  performance  is  measured  against. 

When  the  standard  is  sound,  a  derivative  measure,  labor-hour  variance  (box  5b)  can 
be  computed  to  provide  useful  data.  In  this  case,  the  variance  was  140  labor-hours  in 
both  pay  periods  —  the  equivalent  to  1.75  employees  in  each  period. 

o  A  similar,  complementary  approach  can  be  constructed  based  on  labor-dollars  by 
computing  the  cost  of  the  labor-hours.  The  labor-dollar  approach  may  be  more  useful 
for  monitoring  the  unit’s  budget  performance. 

o  Neither  the  productivity  measure  approach  nor  the  labor-hour  standards  approach 
addresses  the  value  of  the  work  completed.  A  real  situation  might  require  the  agency 
to  prioritize  workload  and  complete  both  types  of  investigations  in  order  to  be 
effective. 

Unfortunately,  there  is  no  standard  solution  to  this  limitation  of  input  productivity  measures. 
When  it  is  known  that  program  activity  during  comparison  periods  is  not  technically  comparable, 
but  roughly  comparable,  substantial  changes  in  input  productivity  measures  can  produce  a  rough 
idea  of  whether  efficiency  has  changed.  One  should  not  be  intimidated  by  the  situation  since 
developing  efficiency  measures  is  an  active  research  topic  in  economics  and  engineering.  You 
may  have  to  consult  with  your  OMB  examiner  to  discuss  an  approach  for  your  program. 

Case  2:  When  a  program ’s  outputs  are  unique. 

Input  productivity  measures  as  output  efficiency  measures  are  not  applicable  to  programs  that  do 
not  repetitively  produce  the  same  things  over  time.  For  example,  research  and  development 
outputs  are  often  one-of-a-kind,  as  often  are  construction  of  special  purpose  infrastructure  such 
as  laboratories,  military  training  facilities,  and  hospitals.  In  these  cases,  it  is  useful  and 
important  for  managers  to  track  how  well  the  program  is  performing  as  it  works  towards 
producing  a  finished  output. 


The  principles  of  sound  project  management  can  often  address  this  limitation  when  efficiency  is 
to  be  measured  with  respect  to  the  program’s  progress  in  accomplishing  planned  objectives. 
Project  management  is  the  discipline  of  organizing  and  managing  resources  and  the  work  of 
projects  in  such  a  way  that  these  resources  deliver  all  the  work  required  to  complete  a  project 
within  defined  scope,  time,  and  cost  constraints.  In  the  case  of  capital  investments,  to  obtain 
timely  information  monitoring  their  progress,  section  300  of  OMB  Circular  A-l  1  requires 
agencies  to  use  a  performance-based  acquisition  management  or  an  earned  value  management 
system  (EVMS),  based  on  the  ANSI/EIA  Standard  748.  Agencies  are  expected  to  achieve,  on 
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average,  90  percent  of  the  cost,  schedule  and  performance  goals  for  major  acquisitions. 

Measures  taken  from  these  systems  that  compare  the  amount  of  the  budget  earned  and  the  actual 
costs  for  the  same  work  (cost  variance)  are  efficiency  measures.  Likewise,  this  type  of  measure 
is  applicable  to  infrastructure  projects  and  other  project-like  activities  that  can  establish  cost, 
schedule,  and  technical  scope  performance  baselines  based  on  sound  cost  estimates  and 
schedules  and  risk  management  practices.  In  particular,  the  approach  can  be  beneficial  for 
managers  of  research  and  development  (R&D)  programs  as  they  can  help  track  schedule 
accomplishment  and  decision  points  as  required  by  the  R&D  Program  Investment  Criteria 
(Appendix  C). 
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GLOSSARY 


Actuals:  A  common  term  used  to  refer  to  data  on  actual  performance  compared  to  goals. 

Annual  performance  goals:  The  measures  and  targets  affected  by  an  activity  in  a  particular 
(generally  near-tenn)  year. 

Baselines:  The  starting  point  from  which  gains  are  measured  and  targets  are  set. 

Beneficiaries:  Those  who  benefit  from  the  favorable  outcome  of  the  program. 

Benefits:  The  outcomes  of  program  efforts  or  direct  assistance  such  as  grant  funding. 

Block/Formula  Grant  programs:  Programs  that  provide  funds  to  State,  local  and  tribal 
governments  and  other  entities  by  formula  or  block  grant,  such  as  the  Department  of  Energy’s 
(DOE)  Weatherization  Assistance  program  and  HHS’  Ryan  White/ AIDS  program. 

Capital  Assets  and  Service  Acquisition  programs:  Programs  that  achieve  their  goals  through 
development  and  acquisition  of  capital  assets  (e.g.  land,  structures,  equipment,  and  intellectual 
property)  or  the  purchase  of  services  (e.g.  maintenance,  and  information  technology).  Program 
examples  include  Navy  Shipbuilding  and  the  Bonneville  Power  Administration. 

Competitive  Grant  program:  Programs  that  provide  funds  to  State,  local  and  tribal 
governments,  organizations,  individuals  and  other  entities  through  a  competitive  process,  such  as 
Health  Centers  at  the  Department  of  Health  and  Human  Services  (HHS). 

Credit  programs:  Programs  that  provide  support  through  loans,  loan  guarantees  and  direct 
credit,  such  as  the  Export  Import  Bank’s  Long  Term  Guarantees  program. 

Crosscutting  analysis 

External  crosscutting  analysis:  A  comparison  between  programs  across  multiple  agencies. 
Internal  crosscutting  analysis:  A  comparison  between  programs  within  the  same  agency. 

Efficiency  measure:  Efficiency  measures  reflect  the  economical  and  effective  acquisition, 
utilization,  and  management  of  resources  to  achieve  program  outcomes  or  produce  program 
outputs.  Efficiency  measures  may  also  reflect  improved  design,  creation,  and  delivery  of  goods 
and  services  to  the  public,  customers,  or  beneficiaries  by  capturing  the  effect  of  intended  changes 
made  to  outputs  aimed  to  reduce  costs  and/or  improve  productivity,  such  as  the  improved 
targeting  of  beneficiaries,  redesign  of  goods  or  services  for  simplified  customer  processing, 
manufacturability,  or  delivery. 

Outcome  efficiency  measure:  A  measure  that  captures  improvements  in  a  program’s 
efficiency  with  respect  to  the  program’s  outcomes.  Generally  considered  the  best  type  of 
efficiency  measure  for  assessing  the  program  overall. 
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Output  efficiency  measure:  A  measure  that  captures  improvements  in  a  program’s 
efficiency  with  respect  to  the  program’s  outputs.  Because  it  may  be  difficult  to  express 
efficiency  measures  in  terms  of  outcomes,  efficiency  measures  could  focus  on  how  to 
produce  a  given  output  level  with  fewer  resources  until  the  time  an  outcome -based  efficiency 
measure  is  developed. 

Input  productivity  measure:  A  type  of  efficiency  measure  defined  as  the  ratio  of  an 
outcome  or  output  to  an  input.  A  measure  is  an  outcome  or  output  efficiency  measure 
depending  on  what  is  in  the  numerator.  Input  productivity  measures  are  relatively  easy  to 
understand  though  their  use  to  make  valid  comparisons  is  often  limited.  (See  Appendix  D 
discusses  some  issues  about  using  these  measures  as  output  efficiency  measures.) 

Direct  Federal  programs:  Programs  where  services  are  provided  primarily  by  employees  of  the 
Federal  Government,  like  the  State  Department’s  Visa  and  Consular  Services  program. 

Long-term  goals:  Goals  that  cover  a  multi-year  period  of  time.  Duration  may  vary  by  program 
but  should  consider  the  nature  of  the  program  and  be  consistent  with  the  periods  for  strategic 
goals  used  in  the  Agency  Strategic  Plan. 

Outcome  measure:  A  measure  of  the  intended  result  or  impact  of  carrying  out  a  program  or 
activity.  They  define  an  event  or  condition  that  is  external  to  the  program  or  activity  and  that  is 
of  direct  importance  to  the  intended  beneficiaries  and/or  the  general  public. 

Output  measures:  Outputs  describe  the  level  of  activity  that  will  be  provided  over  a  period  of 
time,  including  a  description  of  the  characteristics  (e.g.,  timeliness)  established  as  standards  for 
the  activity.  Outputs  refer  to  the  internal  activities  of  a  program  (i.e.,  the  products  and  services 
delivered). 

Performance  goal:  A  target  level  of  perfonnance  over  time  expressed  as  a  tangible,  measurable 
objective,  against  which  actual  achievement  can  be  compared.  Performance  goals  are  expressed 
as  a  quantitative  standard,  value  or  rate.  A  performance  goal  is  comprised  of  a  perfonnance 
measure  with  targets  and  timeframes. 

Performance  measures:  Indicators,  statistics  or  metrics  used  to  gauge  program  performance. 

Program:  An  activity  or  set  of  activities  intended  to  help  achieve  an  outcome  that  benefits  the 
public. 

Program  Assessment  Rating  Tool  (PART):  A  diagnostic  tool  used  to  assess  the  performance 
of  program  activities  systematically  across  the  Federal  government  and  to  improve  program 
performance. 

Reach:  The  distribution  of  program  benefits  (i.e.  program  outcomes,  grant  funding,  etc.). 

Regulatory-Based  programs:  Programs  that  accomplish  their  mission  through  rulemaking  that 
implements,  interprets  or  prescribes  law  or  policy,  or  describes  procedure  or  practice 
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requirements,  such  as  the  U.S.  Environmental  Protection  Agency’s  Mobile  Source  Air  Pollution 
Standards  and  Certification  program. 

Research  and  Development  programs:  Programs  that  focus  on  knowledge  creation  or  its 
application  to  the  creation  of  systems,  methods,  materials,  or  technologies,  such  as  DOE’s  Solar 
Energy  and  NASA‘s  Solar  System  Exploration  programs. 

Results  Not  Demonstrated:  This  rating  is  given  when  programs  do  not  have  acceptable 
performance  measures  or  lack  baselines  and  performance  data.  Specifically,  a  program  that  has 
not  been  able  to  establish  long-term  and  short-term  performance  measures  or  does  not  have  data 
to  indicate  how  it  has  been  perfonning  under  measures  that  have  been  established. 

Strategic  goal:  A  statement  of  aim  or  purpose  included  in  a  strategic  plan  (required  under 
GPRA).  In  a  performance  budget/performance  plan,  strategic  goals  should  be  used  to  group 
multiple  program  outcome  goals.  Each  program  outcome  goal  should  relate  to  and  in  the 
aggregate  be  sufficient  to  influence  the  strategic  goals  or  objectives  and  their  performance 
measures.  Also  called  “strategic  objective.” 

Strategic  objective:  See  “strategic  goal.” 

Strategic  plan:  Pursuant  to  GPRA,  a  five-year  plan  that  includes  a  comprehensive  mission 
statement;  outcome-related  goals  and  objectives,  for  the  major  functions  and  operations;  a 
description  of  how  the  goals  and  objectives  are  to  be  achieved;  and  other  elements. 

Target:  Quantifiable  or  otherwise  measurable  characteristic  that  tells  how  well  a  program  must 
accomplish  a  performance  measure. 
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LIST  OF  ACRONYMS 


Program  Type 


BF 

Block/Formula  Grant  Programs 

CA 

Capital  Assets  and  Service  Acquisition  Programs 

CO 

Competitive  Grant  Programs 

CR 

Credit  Programs 

RD 

Research  and  Development 

RG 

Regulatory  Programs 

General 

BPI 

Budget  and  Perfonnance  Integration 

CDC 

Centers  for  Disease  Control  and  Prevention 

CFDA 

Catalog  of  Federal  Domestic  Assistance 

CFO 

Chief  Financial  Officer 

DOE 

Department  of  Energy 

EO 

Executive  Order 

FAA 

Federal  Aviation  Administration 

FA  ADS 

Federal  Assistance  Award  Data  Systems 

FBI 

Federal  Bureau  of  Investigation 

FFRDC 

Federally  Funded  Research  and  Development  Center 

FOIA 

Freedom  of  Information  Act 

FY 

Fiscal  Year 

GAO 

Government  Accountability  Office 

GPRA 

Government  Performance  and  Results  Act  of  1993 

HHS 

Department  of  Health  and  Human  Services 

IG 

Inspector  General 

IT 

Information  Technology 

LE 

Large  Extent 

NA 

Not  Applicable 

NASA 

National  Aeronautics  and  Space  Administration 

OFFM 

Office  of  Federal  Financial  Management 

OIRA 

Office  of  Information  and  Regulatory  Affairs 

OMB 

Office  of  Management  and  Budget 

OSTP 

Office  of  Science  and  Technology  Policy 

PART 

Program  Assessment  Rating  Tool 
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PMA 

President’s  Management  Agenda 

R&D 

Research  and  Development 

RFA 

Regulatory  Flexibility  Act 

RIA 

Regulatory  Impact  Analyses 

RMO 

Resource  Management  Office 

RND 

Results  Not  Demonstrated 

SBREFA 

Small  Business  Regulatory  Enforcement  Fairness  Act 

SE 

Small  Extent 

UMRA 

Unfunded  Mandates  Reform  Act 
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